upvote
Azure does offer nVidia's secure enclave architecture via "Azure Confidential GPU", but it's limited to a single GPU and a little less than 90GB after overhead, so basically maxes out at Mistral-Large-2 with 4 bit quantization.

This whole thing kind of puts the industry in a bind. On one hand, the big labs do have an incentive to get this kind of thing rolled out so as to avoid open weight on-prem AI gaining traction, and to remove remaining roadblocks to corporate adoption in regulated industries. But rolling it out risks attracting even stronger regulatory responses for them as well (that's even setting aside likely national security/law enforcement pushback), as well as likely being strongly opposed by the AI safety people. There's no one way to square the circle that will please everyone. My guess is the labs will take a third route: continue to lobby against open weight models, thus killing the possibility of on-prem, and then drag their heels on E2EE/secure enclave models except perhaps for military customers.

reply
I'm hopeful that Tinfoil's enclave router, which does decrypt the payload but does so in a verifiably secure manner, can be used to scale to multi-gpu models (for example, Tinfoil does offer Kimi K3.1 Pro).

Because of economies of scale with batch inference as seen with DS4.1 Flash, this is the only avenue for true privacy from frontier models at scale/cost.

In terms of policy, I agree it's looking dire. There's unfortunately just more money/lobbying/mindshare on the 'safety at all costs' camp but I'm still advocating and educating where I can.

reply
We will however need hardware elements that are actually secure. The current generation is trivial for the operator to compromise. https://tee.fail/
reply
deleted
reply