Did they? It says the attacker was able to create unauthorized CVMs (makes sense), able to obtain application credentials (significance unclear to me), and that they rotated their KMS root key in response (which is confusing to me because if they use hardware backed keys then I don't understand how the root key could ever be compromised).
So does this imply they aren't actually providing a proper hardware backed TEE? But that's also confusing because at least if you provision a VM you can have the hardware attest to its state.
The TEEs are real. The problem was caused by software bugs. Running a TEE app (CVM) is more than dropping a binary into TEE. Two problems remain: (1) migration, you don’t want to completely lose the data if the hardware dies, and (2) upgradability, you want to patch the app when there’s a bug without building the state from scratch. So we want to decouple the states from hardware.
Using the hardware key is painful. The solution is to introduce a separate key manager (KMS). It releases keys to the app iff. the app follows a predefined “policy” (code, hardware attestation, etc). We can decouple the states from hardware based on a remote KMS.
This is rather a common design. You can find it in our system, as well as other mature frameworks like Confidential Containers.
As the system grows, a bug was introduced to the policy of how we replicate a KMS’s key material (for redundancy). The attacker exploited it with multiple sophisticated attacks, eventually being able to cheat a running KMS to hand over the keys to a fake one, like it’s a legit KMS replica. As a result, the keys protected by the KMS became at risk.
To mitigate the problem, the legacy KMS was dropped and a new KMS was deployed. We have then rotated all the keys associated with the legacy KMS. A series of protocol and code audit is ongoing to prevent future failure.
(https://docs.phala.com/dstack/design-documents/decentralized... , https://docs.phala.com/dstack/design-documents/key-managemen...)
Actually it looks like a pretty decent solution once the inevitable bugs and footguns are ironed out.