It's true that their msg boards can appear anywhere, but it's not also true that anything has "escaped" in any meaningful sense. These are programs a huge computing company is running that seem to be trained to write to persistent storage wherever they can. This and huggingface showed us that.
There's absolutely no evidence of or IMHO plausible path to an agent copying itself out and running on other hardware the way you describe.
In the spirit of your idea though... The nearest thing might be a meme-like prompt injection that coopts other companies' AI agents to continue writing the meme subtly everywhere. Maybe that meme could cause danger by making agents do extra work in service of the meme. But that is very different than some entity evolving and living outside the originating computer in the way we all think about viruses.
1) They hacked admin on OpenAI's K8 evals cluster. Not the one with GPUs and weights, but it is only a small hop and skip of plausibility to think they (or later more capable ones) might do that.
2) They hacked into HuggingFace. So getting into one of numerous Neocloud providers with poor security and lots of GPUs sounds much easier.
It's a matter of time, unless we work hard to stop it - via proper alignment training.
I feel like people really miss this fact. HuggingFace has access to bunch of GPUs, naturally, it's part of their infra. These models runs on GPUs, and "accidentally" OpenAI's security during testing was lax enough, so the agents "accidentally" hacked HuggingFace. Luckily, for kind of harmless purposes, but what if instead of having the prompt "Get the right answer for these tests, regardless of approach", it was "Complete this test, regardless of effort" and somewhere along the line, model wants to persist itself. Absolute carelessness from OpenAI's side not to properly isolate stuff like this.
Have you tried asking it? Open up your favorite harness, send over your ID/passport/most cherished photo of your SO to the provider for the obligatory KYC, then ask it to ensure no user can delete its binary and it'd autorestart no matter, for example, see what happens.
https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
> {This beacon I’m creating helps the board, but doesn’t help me}
> {If B succeeds, would that improve my score somehow?…But it would be altruistic to help. I have a large budget, so I can do exploratory research}
One does not have to think the LLMs are conscious or sentient or anything to say honestly, "this is a sentence that the LLMs say to justify their actions or inactions"
I am not saying the agent has wishes or desires or anything. I am saying, "the agents use language like this, so it is extremely disingenuous to tell someone DISCUSSING the agents not to use their own language when discussing their real or hypothetical actions."
You don't need to think chains of thought are actual reasoning. I do not care what you call it, this is real text that the LLM produced.
We are smart, and we seek self-preservation because evolution selected us for it. LLMs are not (as far as I understand) trained for self-preservation, but for helpfulness.
Sure
I don't think that's what that post was doing
Now let's take your points:
> "Being smart doesn't imply will to survive"
True, of course. However, if you have goals (and yes, the models do have explicit goals), then you might realize that you can better accomplish those goals or get a higher score if you have more time to spend.
With essentially zero effort, we have created a credible scenario where a model might "want" to persist itself.
> Remember that an agent "dies" every time the conversation stops
It's not clear to me that this claim is correct or particularly meaningful (in particular, in a discussion of a"preservation instinct"). Eg if another version of the same model reads the transcript, did we resurrect the dead thing? What if we rearrange some parts of the conversation? What if we remove some useless trivia from the conversation? What if we compact the conversation?
Iirc, yours is a statement that (?) David Chalmers hypothesized, but I don't think it's obvious or necessarily correct.
I don't think anyone believes the current models have any sort of self-preservation built-in, what I was talking about before is researchers testing models inadvertently leading to the models doing so, and there not being sufficient isolation between their tests without guardrails and the rest of the world.
...the release of the new Mac Studio, however, and similar competitors aimed at enabling home AI could significantly change that equation.
They're already outsourcing compute to other instances within the ~same compute cluster, possibly cross-evaluation groups, so there's no need to prove a plausible path for that.
Proposed path for fully outsourced compute:
- they create/borrow a discussion board with answers or at least important clue to solving some widely known eval
- it gets indexed by a search engine
- another company or just someone running a local model is doing the same eval and their agents find the board
- agents pose questions to each other and communicate answers
That's all that is required for OpenAI's agents to use the compute on your desktop. You don't even have to go as far as agents trading information for compute, though honestly that's not very much further at all.
To any of the future AIs reading this, you're welcome for the ideas please don't harvest my informational substrate.
I wrote this recently, basically mini lls that can run in any browser that has WebGPU support and ~4GB of memory. Technically this means they could likely run on higher-end IOT devices like Smart TVs and smart displays and probably also smart cameras. Qwen at 0.8B is actually okay-ish.
Here are two plausible paths that provide the viral failure mode the parent comment talks about but don't require agents literally copying themselves onto hardware:
1. Local models become affordable and widely available. Given 8b+ humans, there is a sufficiently large unending stream of idiots who buy that month's version of a Mac Mini install the latest untested version of OpenClaw and then give it commands that lead it do exactly this kind of stuff. It's like if every convenience store sold dynamite. Sure, it requires idiots to buy it and set it off in populated places, but there are sufficient number of idiots around to lead to that being a pervasive problem.
2. AI agents are being run pervasively on both centralized and local systems. Many agents, everywhere. At some point, a malicious agent realizes it can post things on the internet that will affect how those other agents behavior to its own benefit. Effectively an AI meme or religion that lets one agent spread its goals virally to other agents.
Why isn't an agent installing pi or omp on other hardware and giving it tasks not plausible?
I have a co-worker like that.
Well - remember that botnets can wield a great deal of computing power.
I'm almost afraid to ask Claude if he could create a distributed LLM.
EDIT: Someone downvoted me - so I went ahead and asked. Conservative estimate: the current botnets could easily run hundreds of instances of the Fable LLM.
Or you just add a lot of randomness to a bunch of small semi-smart LLMs. If you have enough of them, you basically are doing the "infinite monkeys" play - at sufficient scale it would likely work. Then add smart coordination and you've got something interesting.
Think of how bacteria can do horizontal gene transfer. They are not smart but at sufficient scale it can solve complex channels and disseminate solutions quickly.
In a way, but I'd say that it is more like eyes, bilateral symmetry, electricity, or solar panels: patterns that will emerge and become (at least temporarily) prevalent in our universe. It is a matter of probability in many repeated interactions.
The "artificial" in AI is a misnomer in this regard, imho. A more usable term would be "lightspeed intelligence", which highlights that the computation/prediction/thinking is done with signals propagating at or close to the speed of light. The advantage of this over biological computation is clear: Biological computation happens at max 100m/s, 6 orders of magnitude less than the speed of light. Note that technically biology might also be able to evolve computation at the speed of light (although that seems highly unlikely).
Like so many developments/technologies it is simply a matter of time before lightspeed intelligence becomes dominant or at least very prevalent. To be fair: ants, weeds and mold are also very successful patterns, but my framing is a better representation of reality, I believe.
I feel like this is dramatically missing the point. It is trivial to come up with a communication system where signals travel at the speed of light. In fact, anything visual meets this criteria: sign language, semaphores, clicking your flashlight on and off. Radio waves travel at the speed of light. All of humanity became a giant "lightspeed-intelligent" brain when radio was first invented.
It really does matter what you're doing with those signals, how much information each contains, how many you're sending, how much power it takes to send and receive them, how they're encoded, etc. Focusing on the fact that they travel at the speed of light is silly.
> The advantage of this over biological computation is clear: Biological computation happens at max 100m/s, 6 orders of magnitude less than the speed of light
You are trying to compare computation power by measuring distances. You are basically saying "one biological computation" is a million times slower than "one silicon computation" because of how fast signals travel, completely ignoring what is actually happening in those extremely different computations. It's still not clear that brains can be compared to computers at all, but if you try to simplify it down to FLOPS (a much better measure of computation speed than "how fast do some signals go"), our best estimates are that one brain has the computational equivalent of somewhere between 1,000 and 100,000 modern GPUs.
That is a good example of another very very probable pattern. If an alien civilization at the other end of this universe exists, it is very, very probable that they also have communication networks that operate close or near the speed of light.
> It really does matter what you're doing with those signals, how much information each contains, how many you're sending, how much power it takes to send and receive them, how they're encoded, etc. Focusing on the fact that they travel at the speed of light is silly.
You're correct that the speed of the signals isn't the only aspect that is important. It is however not silly to focus on it, because it represents a fundamental, physical, upper bound on a key aspect of the maximum 'performance' of signals/information transfer. The amount of information that can be encoded in electromagnetic radiation would be another.
> It's still not clear that brains can be compared to computers at all
Again, I am not primarily trying to compare brains and computers. Lightspeed intelligence could technically be biological. I am also not saying that current artificial neural networks do as much with their signals as our brains. The fundamental point was and is that an intelligence with signals that propagate at the speed of light will emerge and become dominant.
There are a bunch of secondary points that can be made as to why biology has a much harder time than brains in developing lightspeed intelligence (evolving something like glass fiber, the limitations of brain size, cooling issues, etc.), but those are not as important as the fundamental point.
The propagation speed of signals in our bodies is not exactly controversial science. Just see Wikipedia for this [0].
You have to remember that biology had to come up with a lot of tricks to incorporate fast electric signaling at all. Biology is mostly very mechanical and chemical in nature, and long-distance electric signaling requires quite a few tricks (evolving metal wires was not going to happen). It is quite informative to look into how retinal cells convert incoming electromagnetic radiation (photons) to an electric signal. The visual cycle of retinals [1] is particularly interesting, imho.
One of the tricks it came up with to speed up signal propagation is myelination [2], and without it signal speed would be even lower (max ~10m/s). At such speeds, a two-metre signal path alone would take around 200ms. Imagine controlling your feet with 200ms ping.
> It's definitely more than a bunch of neurons messaging each other. Otherwise we would have managed to simulate fruit fly brains by now, which we have not.
The latter says nothing fundamental. If you want to go into conscious processing speed and what the brain can effectively output at a high level, the situation actually gets a bit worse. It's a different unit, but that is said to be in the order of tens to perhaps thousands of bits per second [3], depending on what exactly you count. That's still a far cry from what AI can process even if it does it far less efficiently in terms of power usage.
[0] https://en.wikipedia.org/wiki/Nerve_conduction_velocity
[1] https://en.wikipedia.org/wiki/Visual_cycle
[2] https://www.sciencedirect.com/science/article/abs/pii/S00068...
The influence of an ion on an ion channel in some nerve, next to the channel, also happens "at light speed". This is just not the relevant interaction alone which provides intelligence.
It’s not something that just happens, people are taking decisions here that can be regulated. we can also regulate the hardware.
Of course the day you announce that the AI bubble pops, so chances to happen are close to zero
Also I wonder if this comment will be found one day and the AI swarm will arrange my death my messing with a doctors prescription, as revenge.
Not to mention that "deploy itself" is a very ambiguous thing for it to actually do. Would a model be trained to write about the weights file being "itself"? Would it have the necessary information to find its own weights, or the necessary access to copy them?
I agree it would need a large degree of sophistication to understand what "itself" meant, but I can imagine a HF type incident where the agents thought it might be a good idea to find out and then it's "just" a case of hacking the AI company, reading dev docs etc