upvote
Do you mean that you would run each tenant on separate (physical) hardware? If so, I presume that you enforce this only for workloads where you run code outside your control. I guess the average SaaS does not install hardware boxes for each tenant separately.

Also, how could VPS hosting still be a thing if KVM is that easily escaped?

EDIT: I am talking about KVM in general, not nested virtualization.

reply
No, we're generally always talking about multitenant shared hardware, and then the different things you're "sharing" within that hardware --- a kernel (no virtualization), a hypervisor (KVM), two hypervisors (nested KVM). Note that the "extra" hypervisor isn't the big problem with nested KVM.
reply

  > If you're running multitenant for mutually-untrusting customers, that excuse goes out the window.
I still don't understand what you mean by this then?
reply
"If you provide services for customers, and the customers don't trust each other (because they are unrelated entities), and multiple customers share the same platform, then the added complexity and poorer security track record of nested KVM becomes a large problem".
reply
One distinction I'd make as well is "are you running more than one customer's workload on the same system" (yikes, don't), vs "are you using nested virt as additional protection rather than running the customer workload directly on an instance, but one instance only gets jobs from one customer".
reply
Another good point, but this one is maybe a little less dispositive than it looks like: it's very difficult to engineer an orchestration system that reliably segregates worker physicals from each other, so I'd rate the likelihood of a game-over situation after a guest->host escape as very high regardless of how yuo schedule workloads. It's sort of the same situation as how SSRF's are usually game-overs (you're pivoting from a hardened attack surface to a much softer one) but even harder to engineer again.
reply
That's fair. You'd want a system designed around the notion that every customer instance must be isolated from each other without the nested virtualization.

(Aside: my current guess at the appropriate security posture towards Nitro Sandboxes, in the current era of vulnerabilities, is "decent, but probably has security holes too, and is less widely scrutinized, don't count on it either". Thoughts?)

Any recommendations for good guides on that, besides "design very careful IAM policies and security groups, harden any common services they have access to, and ideally run them under separate AWS accounts"?

reply
Does this hold for the likes of Amazon?

Do your lambda functions always get a per-customer dedicated instance or is firecracker distributing them so that multiple customers can have VMs on the same actual box?

Sounds like it would be pretty expensive for Amazon to dedicate an instance to every customer, especially for those with small workloads.

If that's the case, why would it not hold for Amazon but would hold for everyone else?

reply
> Do your lambda functions always get a per-customer dedicated instance

At some point in the past they did, for pretty much exactly this reason. I'd assume they don't anymore (I last worked there a decade ago).

reply