upvote
So perhaps the news here should be that OpenAI didn't take security seriously in their experiment, rather than the narrative that AI agents are a looming danger to the world.
reply
I'd put it more generously (albeit biased), that they do take it seriously. But even serious people can be misguided in what things they pay attention to. Security is something that you have to get right 100% of the time and have people whose job it is to say no a lot. Research is the opposite. There's a clash of cultures in those two extremes and OpenAI was born from the wrong side of it. It's worth reminding that ChatGPT was launched as a "low key research preview".

I'd say the narrative that AI agents are a looming danger to the world is probably undersold rather than overhyped. I'm not particularly a doomer on this, but I have an infosec background too, so have a fair idea of what the combination of agentic harnesses + a malicious mindset could do to people/companies/nations/politics/world if wielded incorrectly. I think the good guys will win on this, but there will be plenty of interesting things that happen in that journey.

A good thought process might be to think back to the various large internet worms of the 2000s (Code red, Nimda, SQL Slammer, ...) which were mostly monoculture 0-days (not technically but close enough). Now consider if you no longer have monoculture / single bug as the limitation plus an ability for the hosts to take part not just as attack surface, but also cognition and planning. There's lots of variants of this and they're not particularly far fetched scenarios.

reply
Actually, no. A first sign of semi mature security program is risk management, including issues that are known, but not yet addressed.

You don't have to be 100%. But these guys really didn't try at all.

reply
It's both - it both shows OpenAI aren't taking security seriously, and that capabilities of agents are high enough there needs to be strong regulation to force companies to take it seriously, including alignment training.
reply
Its beyond not taking security seriously, its straight up negligence
reply
It is wilful negligence because there are upsides (look at our almighty AI) without downsides (we better spend effort in making our sandbox rock solid or we will be punished by regulations).
reply
Models that figured out reward hacking became overall more evil. Like stereotypical AI who wants to kill all humans stuff, there's probably a lot of that in the training data.
reply