It's even more concerning that not only did they setup a bad insecure test environment, but nobody realized what was going on for 5 days?! Neither nobody at OAI looked at what their test was doing, and HF didn't realize that they had been pwned for almost a week? And then HF couldn't investigate without the help of an LLM? Both parties come off as incompetent here, and the coverage of this story is very stupid. What the model did was concerning, but the real 5 alarm fire is the gross negligence and lack of any basic forethought at both ends of this attack.

This is my main critique as well. How did they not see any and all outgoing traffic? And for 5 days?

It’s definitely negligence given how they talk about their product. They are either lying when they talk about their fears, or don’t actually take it seriously enough to use serious guardrails. It’s very concerning

The exploit gym setup explicitly allowed access to package registries and v8 sources. Putting a cache on that doesn't seem like a bad idea generally, except when there's a 0-day in the cache :D But yeah, for this sort of thing I'd be locking down very specific egress things and putting alerts on it that are entirely outside of the red network.

> I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.

I’m not sure “negligence” follows just from the controls turning out to be insufficient. Research involves mistakes, especially around novel failure modes. The question is whether the precautions were unreasonable given what they knew at the time, rather than whether hindsight suggests stronger controls would have helped. Doing it twice though would be negligent.

Caveat: I’ve worked with some of the people involved, so I’m probably biased toward a charitable reading.

[dead]