If you cover your phone with an antielectrostatic bag it can't communicate; that is a Faraday cage.
Since people around you will think you are also wearing a tinfoil hard, you had better stick to the phones with hardware switches as sibling comment mentions
I get that anything emitting can be tracked and stuff. I'm looking to take a baby step where I'm at least not having every possible detail recorded and sold. That Phreeli recommendation from another user seems like exactly what I want (paired with other things like a VPN of course).