That's an interesting direction! TEE support would be relatively straightforward with current stack (and it's on my roadmap), so that could be a first step forward.
That seems like a naive take. If any of your local VMs are internet connected and are compromised, side channel attacks could be used to exfiltrate data from other VMs or the host.