I actually trust the Apple Intelligence, when off, doesn't exfiltrate my data.
I too would not want any unprompted access to my files.
At the end of the day this issue is that we don't trust the OS and we cannot easily validate how it is designed to behave.
Baked into the OS implies that it's integral to its operation in a way that the two are fundamentally inseparable. Having a global off switch implies that's not true.
There are other irritating baked in aspects of the newest macos and other recent versions that are arguably less avoidable, like Tahoe's entire UI design, or the Settings app.