upvote
Outside of development and CI, I have yet to hear a compelling use case for nested virtualization.

Agree that the article is downright painful to read though.

reply
I think the use case is that you buy a VM from someone and you also want to run VMs. You wouldn't design your own infrastructure to run VMs inside VMs, but you might already be inside a VM, and want to run VMs.
reply
Oh, sure, there are few limits to the ways you can work around a broken organizational design with a bad technical design. But if nested VMs are the answer then I think you're probably asking the wrong question.
reply
Running a few containers inside a kubernetes cluster that you don't manage, is not that an usual a setup in SaaS things. You don't get full insight into someone else's stack.

That's usually nested, and either cri-o or... KVM.

reply
I mean, nested virtualization allows your containers to implement virtualization-based security. Defense in depth -- more layers for an exploit to have to penetrate.
reply
I personally use it to test virtualization platforms. Not sure if you'd count that as dev.
reply
Outside of the compelling use cases I can't say I've heard any compelling use cases either, now that you mention it.
reply
> I have yet to hear a compelling use case for nested virtualization.

AWS instances are slow to launch; they take whole seconds. Nested virtualization allows running something like firecracker/crosvm/rust-vmm atop a normal AWS instance, without needing a bare-metal one.

reply
If you need as-fast-as-possible guest start, AND you have to be on AWS, AND you can't use any of their native microVM-based services, AND you can't use the bare-metal instances - whether due to cost or some other reason, then nested virtualisation might be the least-bad option. That doesn't make it compelling.
reply
Security primitives which don’t nest are fundamentally a bad API design.

Sometimes that’s justified in the name of making the security code simple enough to reason about, but making it harder to use has a security cost also.

reply
If you're segregating internal workloads, by all means use nested virtualization on EC2 or DO or something. Your threat model is different and you probably have a lot more room to maneuver when a KVM escape is announced, because you have some level of trust in all your workloads.

If you're running multitenant for mutually-untrusting customers, that excuse goes out the window.

reply
Do you mean that you would run each tenant on separate (physical) hardware? If so, I presume that you enforce this only for workloads where you run code outside your control. I guess the average SaaS does not install hardware boxes for each tenant separately.

Also, how could VPS hosting still be a thing if KVM is that easily escaped?

EDIT: I am talking about KVM in general, not nested virtualization.

reply
No, we're generally always talking about multitenant shared hardware, and then the different things you're "sharing" within that hardware --- a kernel (no virtualization), a hypervisor (KVM), two hypervisors (nested KVM). Note that the "extra" hypervisor isn't the big problem with nested KVM.
reply

  > If you're running multitenant for mutually-untrusting customers, that excuse goes out the window.
I still don't understand what you mean by this then?
reply
"If you provide services for customers, and the customers don't trust each other (because they are unrelated entities), and multiple customers share the same platform, then the added complexity and poorer security track record of nested KVM becomes a large problem".
reply
One distinction I'd make as well is "are you running more than one customer's workload on the same system" (yikes, don't), vs "are you using nested virt as additional protection rather than running the customer workload directly on an instance, but one instance only gets jobs from one customer".
reply
Another good point, but this one is maybe a little less dispositive than it looks like: it's very difficult to engineer an orchestration system that reliably segregates worker physicals from each other, so I'd rate the likelihood of a game-over situation after a guest->host escape as very high regardless of how yuo schedule workloads. It's sort of the same situation as how SSRF's are usually game-overs (you're pivoting from a hardened attack surface to a much softer one) but even harder to engineer again.
reply
That's fair. You'd want a system designed around the notion that every customer instance must be isolated from each other without the nested virtualization.

(Aside: my current guess at the appropriate security posture towards Nitro Sandboxes, in the current era of vulnerabilities, is "decent, but probably has security holes too, and is less widely scrutinized, don't count on it either". Thoughts?)

Any recommendations for good guides on that, besides "design very careful IAM policies and security groups, harden any common services they have access to, and ideally run them under separate AWS accounts"?

reply
Does this hold for the likes of Amazon?

Do your lambda functions always get a per-customer dedicated instance or is firecracker distributing them so that multiple customers can have VMs on the same actual box?

Sounds like it would be pretty expensive for Amazon to dedicate an instance to every customer, especially for those with small workloads.

If that's the case, why would it not hold for Amazon but would hold for everyone else?

reply
> Do your lambda functions always get a per-customer dedicated instance

At some point in the past they did, for pretty much exactly this reason. I'd assume they don't anymore (I last worked there a decade ago).

reply
What's the usecase for this? You're already paying for the instance to be up but it's being deliberately under-utilised in case you have to start a VM?
reply
Keep a warm pool to serve customer requests.
reply
? If you don't use it, disable it. Simple as that
reply
> It really should be rewritten by a human.

That does not seem to be what HN has been thinking for months.

reply