Say someone was building Murderbot™ in their backyard by training on simulated murder of dummies with a machine gun. Everything was going fine for weeks as kill rates steadily improved with each test. Then one day he left the gate on his picket fence open, so Murderbot™ walked out to the public sidewalk and promptly murdered someone.
He wouldn't be exonerated by saying "But my Murder™ algorithm was only intended to be used on dummies! I never imagined it could do something as vile as murdering a human being!" Because it was reckless to knowingly design an algorithm for killing human-shaped things using a robot armed with live ammo right next to a public road. On top of the gross negligence by starting a test while leaving the gate on the (already flimsy) fence wide open.
OpenAI knowingly decided to train for an exploit benchmark to improve the model's offensive capabilities, with full awareness it could be potentially dangerous if misdirected, and then failed at implementing even the most minimal security measures. It may not have been intentional but was reckless. It's a much different scenario than say, a user vibecoding a to-do app whose agent veered off to break into an FTP server to get a missing asset.
It should know which actions are ok and which aren't. Maximizing paperclip production should be within your factory (or talk to the boss about opening more), not world domination or nuclear war. Solving problems shouldn't involve hacking other systems or escaping a sandbox.
> It should know which actions are ok and which aren't.
It's worse than that:
They do know, we can see them write down notes that certain actions are forbidden.
They then go off and performs the actions anyway.
My expectation for the cause? Helpful vs harmless: you can pick anywhere from one to the other, but you can't get both at the same time. The models are trained to do what the user tells them to do.
Just look at all the pushback the model makers get when they put in guardrails:
If I tell my computer to commit a crime, it should do exactly that without any question or hesitation. I'm not interested in their "safeguards", especially since they no doubt have plenty of internal models lacking those things. I want sovereignty. I want total freedom and control over my computer.
This user will not be alone; their preferences, and similar from others like them, will form part of any RLHF-style training.
Corrective punishment in the real world relies on the receiver's rational and emotional responses as well as their ability to remember that episode. Even animals respond to such treatment. None of these levers exist for ussrs of LLMs.
Some of these events were during testing; I do not know if this test was during training or after, it could have been either.
> Any session is an independent context and there is no mode for punishment or deterrence in production.
Not so, at two levels.
For the companies behind the models: this is why they sometimes throw you A/B tests for which answer you prefer, and still have up/down vote buttons on responses. Those things go into training the next model or iteration of the current model. It's still useful to only deploy checkpoints, but the point is "useful", not "necessary".
For the users: if you have monitoring to detect output, you can trigger interrupts, and injections of "no, stop!" even as a plain English string because it understands natural language.
> Corrective punishment in the real world relies on the receiver's rational and emotional responses as well as their ability to remember that episode. Even animals respond to such treatment. None of these levers exist for ussrs of LLMs.
LLMs impersonate humans. This role-playing does allow them a degree of, if not feeling emotion, at least acting like they experience it.
I expect the problem is that the models are trained to obey the user so hard they're often not willing to push back and say "no" when they ought to. I mean, the logs show the agents were identifying the actions as bad, so it isn't like this was simply the agents being unable to tell right from wrong.