That's not it. Everyone in Zurich studies French at school, if you speak slowly to them in French, it's pretty likely that they will understand a lot of what you say (if they are familiar with the context of course). Similarly, if you speak slowly in German to a French-speaking Swiss, it's pretty likely that they will understand a lot of what you say (again, if they are familiar with the context).

It's just harder to speak in another language than to understand it, so if you ask someone to speak...

Next time, ask them if they understand French :-).