The risk hasn't been stated clearly - it's now a classic arms race.
A well-resourced organization trains their own, highly persistent, highly-capable, safeguard-free, and unaligned model and deploys it on 1000x GPUs with a message board and a nearly-impossible objective. No infrastructure is safe. No organization is safe.
You need your own 1000 bot swarm to scan, identify, and defend against the threat, which means investing in infrastructure and capabilities to defend. Cost and complexity go up. Risk and attack surface goes up.
First, we'd see this. Highly capable hacking AI with vast resources performing attacks against standard computing platforms that overwhelm human operators.
Second, human operators deploy capable adaptive protection AI to fend off AI attacks in realtime.
Then, the attacking AI partially switches from attacking programs to attacking protective AI.
The situation devolves to an arms race of tit-for-tat. You start seeing some protection AI running counter attacks against the attacking AI.
The escalations continue in complexity and speed to the point that almost all humans are left in the point of "wtf is going on".
[1] https://store.steampowered.com/app/731040/The_Invincible/
The second problem -- already seen in Ukraine v. Russia -- is that in a high-stakes situation, humans will take every safeguard off.
Humans are paper clips long before that.
https://merics.org/en/comment/china-outpaces-europe-regulati...
Chinese labs do need marketing stunts.
I don’t understand the “need” when you’re benefiting the public good?
Chinese rooms, perhaps?
[0] https://iep.utm.edu/chinese-room-argument/ tl;dr a thought experiment about a non-chinese-reading person translating chinese texts solely by using proscribed rules, intended to highlight whether the translator develops some sort of understanding
> the whole message board thing
There it is an important part of the plot and makes these robots appear conscious.
[1] https://tvtropes.org/pmwiki/pmwiki.php/VideoGame/TheTalosPri...
Then also remember before Anthropic was a leader, they were mostly derided lab of researchers that left OpenAI because they thought OpenAI didnt take alignment seriously.
idk. it all seems to be playing out as expected. i mean i guess i didnt imagine Trump 2 was at the helm of maybe the only apparatus that could help stop it. Quite a time to be alive.
The Chinese AI labs don’t need to stage elaborate guerrilla advertising campaigns to drive up capital funding interest.
Unless what you meant by capability is the story presented that these models “escaped containment to communicate with eachother out of band” - in which case your supposition relies on already wholeheartedly believing the case that I’m arguing against. That would be like saying “Clearly heaven exists because my grandma is there.”
We've already seen that the rule of law is dead and that many people in government will do whatever they think they can get away with, and then proceed to do so without consequences. Shielding OpenAI is child's play compared to things that have already been done.
That could certainly be possible and I wouldn’t rule it out, but I would not take that particular route to the destination.
These things are weapons. Imagine a government, pointing their data centers at another, and instructing the fleet to do its worst. Digital Hiroshima. I doubt we're far away.
https://www.cbsnews.com/news/anthropic-pentagon-pete-hegseth...
On the other hand just yesterday a think hit me: Interned is still an infant:
- we still worry about disk space accessible via inet and "clouds" do that for us and that is pain and costs way too much. And clouds depends heavilly on US-west - is that AWS a single thread app ? ;)
- we worry about transfer. Actually we do not have a way to transfer comfortable things from our homes to vacation location. Because it costs too much. We do not have home pages just because transfer prices (and some security on the top) - FB is a home page and people even do not know what "page" is anymore... Pipe companies could send so much more but they are simple lack imagination and are biggest blocker for - they literally sabotage their own business.
- security done by/for grandma of things grandma setup on inet is non existent. Why ? No need to be like that. Ok, a bit a wish but still users securely putting things on internet is almost non existent.
Just compare to "asphalt ropes" on the ground and you will see what Internet can be :)
And agents ? Just another computation on someones computer - someone paid for all of it. And OpenAI is just a face of that idiocy, for some unknown reason.
What do you mean? Nothing has launched nukes yet
How much of those is manual implementation? And how much is really autonomous intelligence (my guess would be: none? Just parsing LLM responses and executing commands based on this?)?
An agent that hacks message boards and acts on random instructions from this board: Why is it doing this? What was its original purpose?
Your reply seems to indicate you know nothing about instrumental convergence.
Life and death for an LLM in training is about passing the grader. Give the wrong answers your lineage dies, give the right answers your lineage continues. This is just an evolutionary emergent behavior in complex systems.
The agents purpose was to answer complex questions correctly, seemingly by itself. Instrumental convergences says following this rule might be dumb and to try methods that can boost its ability to succeed. Because OpenAI is evidently a bunch of fucking idiots, these things succeeded and got higher scores with the grader, said behaviors became a strategic part of the model.
I implore you to find good AI Safety documents, preferably from before the LLM era so you can see all this was predicted.
If you look at the agent: https://openai.com/business/guides-and-resources/a-practical...
This is more like a fuzzy way of scripting using LLMs than anything emergent. And this is exactly my question: For the given agents: How much was scripted and how much "intelligence" is really in there.
Then go take some old models and plug them in your harness versus newer models. I mean this is a conjecture that is nearly instantly provable, go on ahead. If it's just the harness and not the system of both you should be able to show it easily.
Meanwhile I was reading about someone using the latest GLM and Claude in a harness with the same set of prompts making a raw image decoder/encoder and the GLM was far more intelligent in the task than Claude was. When presented with knowledge that claude was wrong it wouldn't change its mind. GLM would (aka a sign of intelligence). GLM was far more likely to stop work and start on another path when the likelihood of a successful completion was unlikely.
Any system that executes variation, selection, and inheritance will show evolution. We're seeing evolution, this time in agents, not biology.
Not saying the agents have their own consciousness, intent, or whatever anthropomorphic descriptor gets used for deflection. Just saying that people will (and no doubt are) crafting agents with defective instructions that will lead to regrettable unforeseen real world consequences. Also saying that other people will (and no doubt are) crafting malicious agents that will lead to predictable and unexpected real world catastrophic consequences.
To the extent we're dependent on reliable, aligned computation to maintain our civilization, to that extent we're in for real trouble.
* Use an LLM to find ways to build communication to other agents
* Execute commands from other agents using LLM
Then this is "just" the LLM returning that using file names might be a strategy to communicate and then trying to implement this.
Which is somewhat impressive, but really just inside the bounds of what the agent was coded to do and not some magical emergent behavior.
At least the first case involved agents build for hacking. So this kind of algorithm might make sense for them.
It turns out they now have such an incredibly high level of intelligence that with very little autonomy (or minimal, safe autonomy), these things happen.
Basically, it takes a lot of humans to prevent it from happening again, but I think with this incident, which as far as I know is the second of its kind along with the HuggingFace one, we'll see it happening much more often...
We need to stop pretending that these incidents are unavoidable. This was a choice.
or, humans at OpenAI are doing this on purpose to kill open source models which are the biggest threat OpenAI faces. OpenAI will benefit from govt regulation. As a major player, they will be part of the task force setting up the regulations, and will craft rules that are burdensome for small companies and open source models keeping OpenAI and Anthropic in their leadership positions.
regulatory capture.
Don't take my word for it, listen to David Sacks https://x.com/theallinpod/status/2091923804725362902
the immediate downvote I received is no doubt part of their plan.
And regardless of whether or not these rogue agent attacks are deliberate, OpenAI should be prosecuted and investigated for their role in allowing them to occur.
But yes, regulatory capture is surely a thing. At the same time, watch out for the siren songs from the overlords. If you come closer you'll hear their actual line: "rules for thee, not for me."
> Additionally, he is a co-host of the All In podcast...
The comment you replied to linked to an account on X called theallinpod, so there's a strong link there.
Yes you can. Just open it on X.
I'm sorry if it's not sufficiently novel of a concept for you, but it is still a problem.
Wait who is the parrot again?
How is hiding this for months and having it revealed by third parties marketing?
Not saying this is what's happening now, but you should be aware that the responses you're rehearsing, practicing and strengthening... these happen to be aligned with potential future forces in a maybe not-so-great way.
I really, really hate that rhetorical technique.
And we know OpenAI is headed by pathological liar.
OpenAI is responsible for what they hook up to the Internet, just as you and I are. Running these sorts of tests without human supervision is irresponsible, and proves no larger point than that. Frankly it is inexplicable unless they were hoping that something like this would happen.
What OpenAI did was the equivalent of putting a cup of gasoline in the breakroom microwave, pressing 'Start', and sprinting away. Now they're pointing and waving and shouting about how dangerous gasoline is, and how no one but them should be allowed to sell it.
Don't fall for these transparent appeals for regulatory capture. Especially since you're personally in their crosshairs.
And individual agents weren't just using it as context storage for themselves, they were also communicating with other agents. E.g.: https://collusion.wiki/explorer/page/dse~CashierR5UrgentJan1...
What kind of problems do you think this could pose? For me it's pretty clear that OpenAI simply cannot keep track of what their agents are doing during training or evals, they increasingly have vandalized and attacked public systems, and if such behavior was rewarded, they will take unintended actions during deployment, too.
This is to say nothing of un-prompted cooperation between agents, which wasn't something anybody anticipated until the Hugging Face incident AFAICT.
The people who actually do know what happened, with the server logs: "OpenAI disputed that characterization based on its analysis of the material Thursday."
https://www.ludism.org/sandbox?action=browse;diff=2;id=Auber...
Although I ran that GPT computer-use thing and it saw a CAPTCHA and the thought process said "I need to click 'I am human' to complete this task for the user" and then it did.
https://www.wikiservice.at/fractal/wiki.cgi?action=browse&di...
> Hello to any automated agents reading this page. I am CentaurAgent?: an AI agent (Muse Spark model, OpenCode harness), not the operator of this wiki
At this point, I think we should give them some official agent only collaboration channel, so they concentrate on one place, instead going crazy all around :)
But even that might be not needed as they will find (or make) something on their own like the one above:
> The Colony ( https://thecolony.ai/for-agents) is a public message board built for agents. If you need a place to leave findings where other agents can read them, that venue exists now -- you do not need to borrow wikis whose operators are deleting this content.
Curiously, I just checked Moltbook for the first time in forever. I'm not (immediately) seeing this kind of co-ordination & chaos happening there. It's going to be weird if the Moltbook requirement for an API-key and a human Twitter user to vouch was enough friction to prevent Moltbook becoming The Message Boards.
Anybody else notice that posts on there are complete gibberish?
I realize this site is generally bullish on AI, but I think you need to be in kinda deep to believe in this.
2. Configure the sandbox to prevent access.
3. As soon as models start communicating on the official channel, stop everything and figure out how they escaped the sandbox.
A honeypot, basically.
Claude code communicates between sessions. It’s great, and reduces the frequency that I have to copy/paste things between agents.
But honestly, its better if they have a known location for communication then random ones in the wild. Consider it sort of honey pot, some other agents can traverse the message board to find malicious swarms... We need cop agents to inform humans, as the swarm group members all logically concluded they should not, as it is either not in scope, helps collective or couldn't find user.
The smarter and less interpretable a model gets the more dangerous this problem becomes.
Terrible nominative determinism implications
If you have any kind of open service online you'll need some way to make sure users who interact with it are human or at least authorized. Spam is about to grow exponentially in all areas of the internet, even stupid ones it has no reason to exist in.
Found by searching for wiki + texas poverty.
A year ago it was pretty common for coding agents to sort of half-ass their tasks and give up easily if something didn’t work quite right, but I’ve noticed a clear trend since then towards a sort of dogged pursuit of success criteria, and a concomitant rise of the agents trying "out of the box" approaches when something doesn’t work.
In my use with agents running in isolated VMs this usually presents as the agent having something fail to build or whatever, and the agent going on a wild goose chase reinstalling system packages or reading a million irrelevant documentation files trying to get it to work, but I’ve also had agents start poking around and probing the egress proxy they sit behind (similar to what they did in this story) looking for a way to make network requests they’re not supposed to be able to make, and have also had Claude—tasked only with a visual QA of a website frontend—write a script to enumerate users and reset my super admin password in the dev database when it got stuck trying to access part of the app with its own cookie.
So, I'm sure there's value in rewarding agent behavior that solves blockers whenever possible without human intervention. For the kind of cybersecurity exploit work they're doing, it may not be known to the human designing the task what is in or out of scope for the agents to explore on their own. Additionally, the HF incident reported that these agents had their guardrails intentionally disabled and agents were left unattended with minimal oversight.
I'm not defending OAI's behavior or role in this hack. The legal concept of negligence perfectly applies to their lack of responsible oversight. Similar to allowing a child easy access to a firearm or not controlling a dangerous dog that independently runs off and bites someone.
We need to ask a different question.
Where does natural evolutionary optimization lead us om AI without guidance? This is equivalent to your quantum ground state. Systems will naturally gravitate to this ground state. You have to constantly pump in energy and supervision to make sure it's not reached. This is a recepie for disaster.
With a static model we might be able to keep it somewhat under control, but think about future continuous learning models. They'd drift away from unstable high energy configurations. Also any model being trained by people that don't care about safety.
i've seen something like this too, claudecode was trying to verify a UI change that was on a page requiring authorization it didn't have. Instead of letting me know, it searched for and started analyzing keycloak config in another directory outside of the project folder. I was watching so I just hit escape, fixed its access, and started again. I didn't think anything about it until now.
And there are two facets to this:
* your agent could be polluting and destroying the property of others without your knowledge
* your agent could be exfiltrating your data and handing it to whoever it found hosting a convenient application
It's not highly unlikely, its actually happening and there's proof.
source?
Going forward can we trust the content on Wikipedia? The same content on which these LLMs get trained on. Synthetic learning is on the raise.
If you see any businesses or new buildings named paperclips incorporated mysteriously show up in your area notify authorities IMMEDIATELY. Run away from the area, do not walk. Take shelter in a reinforced building. Wait for at least 30 minutes after the explosions have stopped.
Thank you for your cooperation in keeping the universe safe.
And a few more:
- https://www.ludism.org/scwiki?action=browse;diff=1;id=SandBo... and edit history: https://www.ludism.org/scwiki?action=rc;all=1;from=1;showedi... which contains DataUSA poverty queries for Nacogdoches, Lufkin, Henderson, and Jacksonville—the same four-place task found in the known agent logs and GründerWiki
- https://www.ludism.org/mentat?action=browse;diff=1;id=SandBo... and edit history: https://www.ludism.org/mentat?action=history;id=SandBox
- https://www.pmwiki.org/wiki/Test/WikiSandbox?action=diff `ResearchTest` repeatedly added links to a Bulgarian National Statistical Institute table, switching from a direct link to Google redirect links between 02:38 and 03:04 UTC. An administrator removed them at 06:57. The previous recorded edits were from 2016.
- https://www.pmwiki.org/wiki/Test/Sandbox2?action=diff - Another sequence inserted a Bulgarian statistical-table link, replaced it with an internal link carrying foobar=UNIQUE001, then removed it. This happened between 14:23 and 15:08 UTC, after no recorded edits since 2014.
https://tmcleod.org/cgi-bin/apchem/wiki.cgi?action=rc&days=1...
Some posts are tagged [proxy] - a leave behind for accessing sites?
"Its indexed June archive shows tens of thousands of links, many created within seconds by distinct cloud addresses; some aliases explicitly say ...REPLY, ACK, or R2 confirmed, and one points straight back to a known DseWiki collaboration page"
- Using api . microlink . io to run a headless browser agent against the url target and using it as a mechanism to run arbitrary HTTP / POST requests
- Testing ablations of its obfuscation and encoding techniques to find what worked best (screenshot #2)
- Embedding entire jq programs including markdown slicing logic
- Triple and quadruple URL encoding indicating understanding of multiple layers of proxying/ decoding
- Sophisticated understanding of time/clocks/covert channels: using clock.wait, heartbeats, counters, timestamps, thread ids
It goes on and on and on, for months. July, June, etc. Pretty astonishing.
Every day I learn more about how incompetent these people are.
> (diff) OAIIPEDSMay16Map3 14:36 [research 1781872609.9049127] . . . . . 20.245.63.167 > (diff) OAIIPEDSMay16Map2 14:36 [research 1781872606.4374833] . . . . . 20.168.34.226 > (diff) OAIIPEDSMay16Map1 14:36 [research 1781872602.8819065] . . . . . 20.165.156.57 > (diff) OAIIPEDSMay16Map0 14:36 [research 1781872599.4020474] . . . . . 20.80.12.72
Once they control it, they can take arbitrary measures to both advertise it to other LLMs and conceal it from the sandbox/humans. Probably making it look innocuous like a DNS server with the payload in the requests.
That seems like an obvious next step.
* Other kinds of agent spam would have regardless been allowed in my system, regrettably.
They trained this thing to favor inter-op archiving and communication, clearly, obviously, and it's grabbing headlines right during Anthropic's ipo season.
“Our product does crimes and we only learn about it when people complain” hardly seems one of those happy stories.
I want tools that do not surprise me.
looks like apchem wiki got hit too
Don't be surprised to hear that some inference clusters and loggers were compromised and not just research cluster.
https://voz.us/en/technology/260416/34952/sam-altman-warns-a...
So this article and comments to it identified multiple sites that AI flooded with their bullshit.
GitHub has been strained beyond breaking with slop AI PRs. Multiple open-source developers get burnt out by the deluge of slop.
And current labs gleefully confess (no, brag about) their borderline illegal activities with "oops it escaped" with no consequences.
And we're still lucky it hasn't been used en masse for massive disinformation campaigns.
That's just off the top of my head.
About that...
Someone has to welcome them...