I've actually observed similar behavior at home.
I have a k3s cluster running at home. I asked an agent to check some stuff as a normal user but I had kubectl access to the k3s cluster.
Part of the research, I'd allowed access to run kubectl commands for spinning up test containers. However, when the agent ran into something that needed sudo, it realized it didn't have access there so it immediately used k3s and mounted a localpath into an ephemeral pod to gain access. Sort of horrifying how fast and natural it was for the agent just checking my network (it found the problem fyi).
None of this is very exceptional other than the fact that an agent doesn't have any sort of qualms using any route available to elevate permissions.
If they can bypass it there is no security and the security was flawed all along.
Good security is extremely hard.
If you look at the 90s + 00s, everything was moving towards unified systems, things like small talk, winforms, spring, asp.net, etc. were moving everything into the IDE, you used one language, one framework, one build system. Then people started adding javascript, but even that was getting semi-unified as people coalesced on jQuery, jQueryUI, etc.
Then something happened in the late 00s/10s, and suddenly we had SPAs and noSQL, then microservices, then k8s and now we're here, in what is a mish-mash of 10/20 different systems with 10/20 different attack surfaces.
As my own off-the-cuff guess of what happened, I think perhaps people tried to apply the Unix philosophy, but without a central committee keeping everything aligned it's really not worked.
Serving an interactive page that stores data over sessions should be a trivial solved problem at this point, and instead we've somehow made it where often the scaffold is vastly more complicated than the actual business logic.
FLOSS killed market for end-device software. Cloud+SaaS neutered FLOSS (the code is running literally out of your reach, so may as well be open and free, for any good that'll do you).
And this does actually connect to the security discussion, because despite the apparent belief that "security" is an unqualified good, it is actually just a mechanism of control, and whether or not it is good for you, depends on who is doing the protecting, and who are they protecting from. Very often these days, that threat actor is you.
Perhaps it would be helpful in these discussions if people mentally swapped "cybersecurity" for "police" or "military" or "humor of bureaucrats with power over you" - then it would be more obvious just how important it is to distinguish when you're being secured vs. you're being secured from, vs. accidentally finding yourself in the gears of the security aparattus.
Security by obscurity is pretty useless against people and ai that are smarter than us.
Of course it’s exceptional agent capability when compared to all of history previous to one week ago.
Like, I know everyone here obsesses over AI and uses and follows it very closely, but come on guys. Yes, it is wild that these things are this good. This technology is still brand new. It could t do basic maths a year ago.
Sure, the OAI team was negligent in various ways, and they should be held culpable. But that doesn’t detract from the true black magic that is these modern models.
We know how these things work.
They had the guardrails off and gave it a task and it did it in a roundabout way because these things have no ethics or judgement.
If you did this you’d already be in jail.
These things often look obvious and simple after the fact. Finding the weakness in the first place is the hard part, and that’s what makes the agent’s capabilities interesting here, especially at scale.
But I guess in the actual world we live in, this is just something that happens, and we all shrug and move on and hope that nothing worse is going to happen tomorrow.
Seems to me you could say this about all enterprise adoption of "AI" since 2023.
Licencing fee structures and human laziness motivates single instances. Feature growth results in multiple independent services in the same system. Delivering features quickly motivates lack of rigor, a complete absence of systematic security testing.
On the client side, valid fears about supply chain security are painted over with scanning so they can keep using nodejs and PyPI and moving quickly. Tools designed for humans are pressed into service as AI interfaces, but without human restraint they need rethinking.
A whole industry has been built on the idea of worrying about downside risk if it happens, and just not being the slowest in the pack. No one thought it could happen to everyone at once.
And for my fellow engineers who would think "oh no, they wouldn't do that". Remember that these places employ the apex predators of software engineers. They've already been proven in court that they are very capable of this with all the copyright violation they had to do to get the training data. THESE PEOPLE ARE NOT LIKE YOUR COLLEAGUES.
"Btw don't turn the planet into paperclips"