What I think we're really after though is something like gVisor, where the guest program is completely isolated from the host kernel, and the daemons that allow the guest program to reach the outside world are themselves highly locked down by the host kernel using technologies like seccomp-bpf and namespacing, on top of whatever constraints and validation they apply on their own. While nothing is foolproof, this feels like, if done carefully, it would give you a very good layer of isolation that would be extremely challenging to bypass. I reckon that the sandbox would cease to be the most interesting attack target in a system like gVisor, since in any complicated system, there will probably always be some lower-hanging fruit. (And of course, TinyKVM seems to be basically in the same wheelhouse. None of these solutions are designed to run GUI software, though I reckon it probably could be made to work.)
I think it should be possible to pass /dev/kvm as an open fd to daemons like kvm server and mark it as non-inheritable. As long as the vm is in a subprocess it would be okay I guess.