You can do what you're describing but it's not the model Google is expecting you to use, and you shouldn't have to do that.
It seems what happened here is that some extremely overzealous PM, probably fueled by Google's insane push to maximize Gemini's usage, decided that the Gemini API on GCP should be default enabled to make it easier for people to deploy, either being unaware or intentionally overlooking the obvious security implications of doing so. It's a huge mistake.
Like deciding ATM cabinets should be default open to make it easier for people to withdraw cash.
No, there must be more behind this than overzealotry.
Artifical Intelligence service design and lack of human intelligence are highly correlated. Who'd have guessed??