You can still ask. If you're too afraid to even do that, then the alternative is to be guarded during the conversation. But you can choose which path to take.

Really? In my everyday people always ask "Is it okay for my/our notetaker to join the meeting?"

Might be a German/European thing though...