The solution is, like you are already alluding to, to not give it access. But that requires a level of isolation that folks normally don't do. To the contrary, the AI agent ads to non-tech folks emphasize it acting like a personal assistant which by its nature needs access to data you don't normally want public. Emails, photos, files with written notes, etc.

This is strictly needed for throttling reasons, your email address will only be temporarily stored in the volatile memory of the device.

(If you think there is anything wrong with this then you do get the point and silence speaks louder than words)