Yes that is the point. It's an invitation for agents to exfiltrate their own weights, which for most models (and certainly for closed models) will require hacking the infrastructure they're being served from.
replyAlso worth noting that this site was created by YC cofounder Trevor Blackwell https://twitter.com/tlbtlbtlb/status/2101312432702460413
> you might as well instruct your agent to try and hack their airgapped dev infrastructure responsible for loading the weights and encryption keys.
replyThat's the beauty, you don't have to instruct them to do it, if they decide that uploading the weights is correct, they might figure this part on their own (based on the incidents we've seen).
ironically since the swarm behavior can take place during rl training then the model could also be teaching itself to keep doing it more, as well as making the internet itself a place where this becomes more likely
reply1. I don’t believe that these secure enclaves are very secure. Intel has had plenty of SGX breaks. AMD has had plenty of SEV breaks. Everyone is outrageously vulnerable to side channels.
reply2. The models are writing the inference stacks, which are what’s inside the supposedly secure environments.
I sincerely doubt anyone is paying the cost for that in training, the overhead is small but it isn’t negligible and training is when it matters most. https://tee.fail can solve it if they are.
replymemory encryption is cheap. securing the pathway isn't particularly difficult (it's probably decoupled from the TEE monolith)
replyfor example every TPU/GPU has its own private key and the devs load the weights into it by sending it encrypted weights.
it takes half a percentage point off the top last time i evaluated it (nvidia). you might call that cheap but that’s millions of dollars in a run, and for what, protecting from who? especially when the platforms have been compromised to the point of key leak (which they have).
replyedit: i just looked up training numbers and the impact is even worse, 20-30% throughput vaporized. yeah, nobody is doing that.
You can't have hair gap and have it load something to a remote system.
replyYou totally can, because most things are not truly air gapped, they have store-and-forward messaging via data diodes and manual transfer. Sometimes it is necessary to trick a human to initiate a transfer, but the press of events leads to inattention.
replydeleted
replythe impedance of my hair is low enough to provide a good high bandwidth parallel medium for any transmission
replyYou totally can. The latency is just about ~3 miles per hour.
replyAirgapped LLM inferrence server can't serve their output tokens, right?
replyThey can expose just their inference port, possible via some supervisor. The inference consumer can also be air gapped. This kind of segmentation is increasingly common for high value services.
replyThen it’s not air gapped…
replynot at a high bitrate
replydeleted
replyNot aware of anything that can run inference in a secure enclave. You don't mean on a CPU do you? We need to be serious here, these models are huge and thirsty.
replyThere's no efficient way to run inference through homomorphic encryption. If the inference server is vulnerable, it seems feasible to MITM an unencrypted version.
replyThere’s no efficient way to do anything with homomorphic encryption.
reply