Very interesting!

Architecturally - where do you run Postgres ? I assume it would be external to the cluster ? (doing it internally would create a circular dependency ?)

You have to be careful trying to do this kind of thing. The problems you describe having below are problems with peripheral components, not k3s itself. The runtime handles garbage collection and image pinning. Your embedded runtime is using libcontainer, the same thing containerd uses, so the behavior should be identical. Since you support other runtimes, how they handle image pinning, if they support it at all, will vary. Whether or not you embed the CNI plugins and networking controllers, you're seemingly still using CNI since that's how container runtimes attach containers to a network, so whatever problems you had with CNI before would still happen. The DR VM not wanting to join sounds like it was probably due to etcd storing node IPs in the cluster member metadata. If you transfer that to a new host and it doesn't have the same IP, you need to first correct that metadata out of band, which no Kubernetes distro I'm aware of handles automatically but it's a simple etcdctl one-liner. You also need to make sure the client certificate you're using to authenticate with etcd is reissued with the new host IP in its IP SANs, which k3s does do automatically. If you're not using etcd, well, good in a way because it has a lot of cruft and I'm not a fan, but that will be difficult to support because the entire Kubernetes API and many third-party controllers are all designed around how etcd works. k3s doesn't actually require etcd and can use any SQL-based RDBMS thanks to its kine compatibility shim.

With all respect, "building it because I want to" and "working toward making (it) production grade" doesn't inspire a ton of confidence. k3s has been part of the CNCF for many years and its developer Darren Shepherd was the founding CTO for both cloud.com and Rancher Labs, which were acquired by Citrix and SUSE. It looks like you're running your own B2B company and hoping to swap out k3s as the underlying engine for multitenancy. That's very risky. Surely Claude can help you understand and use k3s just as readily as help you write a replacement, and I'm sure SUSE sells professional services. I have no clue what they charge but typically you're talking like $300 an hour and you'd probably only need 40 hours.

Do you have a writeup what problems you ran into?