The SVP is not considered "moderate". They are a far-right party. The fact that they are wide spread and gather a lot of votes does not make them "moderate".

Source: am Swiss

As an anecdote re Germans: A friend of mine did an Auslandssemester there and was surprised to see "No Germans" signs for some of the housing options. Always makes me chuckle as an example how "relative" xenophobia is.