The original ExploitGym used network-layer isolation. OpenAI downgraded that to trusting the repository server's settings. Giving the agent that kind of attack surface seems like a pretty foreseeable problem. I don't think it was intentional, but it's not a stretch to say that they aren't good enough at this job.

> It’s reasonable to assume that a bug was fixed when reported.

You must not have reported many bugs then. If you don’t see release notes or confirmation from a trusted source, you should assume it’s still a problem. See Microsoft and their “It’s not a vulnerability just a design choice :)” defense

If your CEO is going around talking about how your product will "most likely lead to the end of the world", people are right to expect you to be pretty careful in what you're doing. OpenAI allowed bidirectional communication across security domains for over a month before discovery. Even after it was discovered (and not completely fixed), they didn't set up monitoring able to detect attacks against internal or external services, which went on for further weeks.

Monitoring for sandbox escapes (and sandbox design in general) is easily done these days.

It has nothing to do with nice. These are bare minimum standards we should expect from “big companies” with near infinite resources.

Their constant drum beating about the cybersecurity capabilities of their own models only makes this worse because they’ve displayed that they understand the risk and still did not practice due care.

That’s the definition of incompetence.

Also, this is a (semi-intentionally) evolutionary process where any communication medium that was visible to monitoring would disappear.

So by definition the only ones that appear are the ones that are not visible to monitoring.

If:

1. you have something that can find RCE's in leading commercial systems

2. its training gives it drives to communicate successfully with its peers

3. you are a leading commercial system

4. you run it ~10^10 times (the number they gave in the talk)

...it's really hard to have strong certainty up front that it's not going to end up successfully communicating with its peers.

[dead]

People seem to think LLMs are sentient and not tightly coupled to server infrastructure OAI are in full control over. They literally have every single message sent and action taken, if they can’t monitor that during test runs they should find another job.

It’s like running a benchmark and forgetting to tune it and then wondering why production is running slow.