that requiring access to a privileged network is NOT a sufficient security boundary! Someone you don't like always somehow ends up getting access to the network.
Security is an onion, you have lots of layers because layers peel off and fail all the time. With your current setup a single foothold gives an attacker all the lateral movement they need for full exploitation of every single thing you own blowing past the other layers.