(www.csoonline.com)
A huge chunk of their revenue comes from US-based enterprise customers. Earlier this year, in an effort to become profitable, they tried raising their rates. Those enterprise customers responded, in part, by ramping up efforts to migrate to open weight models. If those efforts are successful then OpenAI and Anthropic will lose their one remaining plausible path to profitability. The US banning open weight models arguably keeps that path open. I don't personally buy that story, but I do believe hedge funds and institutional investors would. That could be all that's needed for a successful IPO.
As for cybersecurity implications, long term impact on domestic AI industry or the US tech industry more broadly, geopolitics, etc., I don't think OpenAI's or Anthropic's leaders are any more worried about that than your average termite is about the structural integrity of the house it's eating.
You forgot about Ads. The one true endgame for every tech company.
Easy peasy. Shouldn't be too hard at all.
> The US banning open weight models arguably keeps that path open.
They stole the collective data of humanity as a whole and now sell it back to you in a rent scheme. Open source will be starved of data and compute.
You're supposed to become dependent on that "service" and slave away at paying for it, while being able to earn anything only in conjunction with it.
Note how difficult it already is to get any up-to-date documentation on anything capable of generating actual market value. A monopoly on competency ensures maximal profits.
AGI, I think, is an illusion: you need a whole lot of specialized bots. Heck, when I query a bot wrt something I know nothing about, I'm blown away by its ability to inform me. But when I ask about something I DO know about, its answers are half-assed. Heck, even when it can access the internet, it avoids all the sites that have paywalls, so it sends me to little reddit to conversations that are entirely untrustworthy.
They make their money on token burn. They can’t do that if they’re competing with something that’s 80-90% as capable, but 1/10th of the cost (with the option of deploying it yourself for “free”).
Wealth accumulation via scare tactics is a classic move of defense contractors and AI firms are quite capable of running the same playbook.
The goal is power. Having 30 million of personal wealth is enough to retire in luxury and travel the world. They could all have this much right now. That's not what these types of people want.
Having 30 billion gives you the power to order legions of people around, buy and sell politicians, flood ad markets during elections, all with your couch cushion money. That's what motivates these types.
As is a 2000$ gaming laptop can a run a basic llm which is probably good enough for most basic programming.
Most companies are going to be forced to use “safe” LLMs and pay by the token. Maybe for half a million Anthro will let you host on site, but even this will face usage fees or caps.
At this moment in time, it seems frontier market needs regulation - market for open does not actually.
Oh.. they dont want regulation on that market? Boohoo!
Other than China the rest of the world has dick for capacity to run models. Even China is at 1/10th the US is and is at to stay for a considerable amount of time.
Don't get me wrong, I want a world of open models, but I don't see us living in that world in our current trajectory.
Not supporting closed models was initially an ideological choice, but it became a technical choice as the context management approach I've implemented requires the full reasoning to be available. All of the closed models are now keeping the full reasoning server side only.
OpenCode-go is a good personal subscription choice
I use both
Both have taken huge amounts of investment money, and those shareholders will come knocking for their returns at some point.
They make money on token use, but that’s hard to do when China is launching models that are 80-90% as capable, cost 1/10th, and can be deployed for “free”.
I imagine they’re also pissed that most of the Chinese models are distilled from their own (the irony being that they scraped the entire internet to train their own ones).
There is a long, convoluted set of legal arguments that are going to make the weights of a model "facts", rather than "code" making it even harder to "ban". Furthermore having them not be "facts" is going to open up a whole other legal argument about what they are and what the attribution is.
I suspect that they are fishing for a case, so a judge can set some precedent.
This is a silly meme. Who wants to pay for evil models that go rogue in your company infra?
Maybe Sam Altman is getting worried that they've raised more than $150 billion and still have net losses and thinks of it as the only exit? I don't know, but someone else provide an alternative explanation as to why a company would work so hard to turn the public against AI.
They're looking for regulatory capture. They've spent ${bignum} on AI and are possibly looking to have the government say "You can't deploy an AI model unless you've spent ${bignum} on it and done all these steps to make sure that nothing goes bad."
At that point, only the existing AI companies that have spent that money can continue forward until someone else spends ${bignum} on it too.
Another aspect of it is if there are regulations established, it may be used to reduce liability ("we followed all the steps proscribed in section whatever, the company is absolved of liability for its actions and that falls on the user instead.")
Well, as people say over and over again, they've got no moat. The actual LLM models are destined to be commoditized over time, and the real money is in the hardware to run them on. The endgame winners are going to be Nvidia and AMD, not Anthropic or OpenAI.
So what's the typical playbook for large firms seeking to insulate themselves against market compensation? Push for regulation. If there are no natural barriers to entry, create artificial ones. And how do you get regulation passed? Present yourself as the target of the regulation, rather than its beneficiary, and start publicly doing things that make others think you need to be reined in.
These stunts, especially from OpenAI, aren't just trying to hype up the power of their own product, they're trying to hype up the perceived threat of the entire category, so they can provoke political intervention that they can comply with, but few others can.
See Meta et al complaining about Section 230 of the DMCA. Meta would be entirely able to moderate in an environment where mass scale moderation/censorship would be required to prevent a host's liability for UGC. Almost no one else would be able to do so.
That means no upstart could possible come along to dethrone Meta as they did with MySpace. It would also be impractical to run a FOSS social network (or forums). Meta could literally just have bots sign up on a FOSS social network until its DAU count ran over the statutory minimums to be subject to UGC liability.
Regulatory capture is a de facto ban on competition in a market.
The private market price is irrelevant - supply and demand determines what they will be sold for in liquidation.
Quite frankly Im bored of this drama. Get it over with already - go IPO, show your financials, find a path to being a healthy going concern or die! Simple! Nobody has a divine right to exist.
If the US govt takes over, existing employees and shareholders will get screwed.
If any of us did a tenth of what they did, we would be already in jail.
Any model with a big enough prompt can and will drift. The model in question was a highly persistent one which means even more opportunity to drift off the normal path, and this really showed in unanswerable questions.
We can all believe that models do things unreliably all the time, but when that lack of reliability turns into hacking it suddenly can't be true?
The more extreme your opening demand, the greater the likelihood that the eventual equilibrium point will be closer to your actual target.
And in this case the actual target is to entrench the position of the current market leaders, and then effectively close the market to new entrants. That's itself a very extreme position, so the opener has to be even more extreme than that.
The denial phase is still with us.
I think the idea has some merit, but it only buys time, as people will soon be able to train dangerously powerful AIs in secret. I think the government needs to force the labs to solve alignment before developing more powerful models. Then the question of how to counter a poorly-aligned, rogue model arises; it's guaranteed somebody's going to let one loose.
And it doesnt really matter if it was actual terrorism or false flags, the end result for either is the same.
that this article is no.1 on HN right now shows that media could successfully use this framing when reporting on security incidents. i.e. readers will generally accept that framing and engage with the story (i.e. drive views/impressions).
so "the more evil stuff these agents do the more capable and impressive it makes them look" is not true in any deep sense. that there are perverse incentives driving the media to choose that framing instead of the framing espoused by this article is real, and probably far more perverse than you're acknowledging. i.e. "trace the money" type of perversion; just offhand I know that OpenAI has purchased at least one media outlet that reports on AI happenings: of course that outlet is going to run the "OpenAI's LLM is so powerful it escaped containment" narrative. that this effect might exist across much larger scales used to be a reasonably controversial claim: just observationally, that idea seems to be received with less skepticism with each passing year.
There was all kinds of science fiction that showed a world of robots and AI, but nothing about the transition. We have an idea what the future looks like, but not the path in between. If AI gets powerful, then there is some point where that power leaks out in a series of events. Would we expect it to look any different than it does now?
For example, parents are routinely held liable for their sentient children's actions.
Corporations are routinely held liable for their sentient employees' mistakes.
The proliferation of this completely nonsensical theory across HN is disheartening to say the least.
AI sentience is not the nonsense theory. If it's clearly nonsense then don't argue with me about it, just publish your paper and retrieve your Nobel. I'll read your argument then.
The nonsense theory is that people are pushing anthropomorphization to limit liability. There is zero evidence for this theory and the underlying assumption (that a sentient model would reduce liability for its lab) is completely incoherent. That's not how liability works.
One is liability over action that harm individuals or smaller populations like a company. There is no escape from that.
Then there is the second liability of creating something that escapes in the much more traditional sense. If I release a virus that kills 90% of humanity, a I'm liable, but it's a bit past that point now. Isn't it?
Instead it's about risk management and preventing labs from taking bets that have unlimited liability for everyone.
They can get their publicity while paying a one trillion dollar settlement.
https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che...
Though I suppose it is the B in DABDA... not excited for the societal slump into the second D, much less the wave of people who have yet to leave the first D to enter A :(
These are people playing mathematician, biologist, etc. They see big words and just assume they're meaningful.
It's just a weird techno cult at this point with a mostly functional product.
We're also at a time with authoritarianism is on the rise.
Not hard to see how this can impact those workers, potentially ruining their lives.
For technology like this the entire selling point of it is how powerful and capable it is
Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to
I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising
You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”
Truly insane take to think the labs being concerned about AI Safety is actually a marketing ploy. Deranged stuff.
I would encourage reading the actual report, which is explicit that a previously unknown exploit was used to escape the sandbox:
“Agents exploited a separate, previously unknown vulnerability in Artifactory to move beyond their sandbox environment and compromise the Hugging Face environment.” — Section IV.A, p. 9 in https://github.com/zeke/openai-huggingface-incident-research...
This name makes me less surprised :)
"The reason for the apocalypse, in the end, will not be that surprising "
Yes, if someone directed them to hack external systems, that would (in my opinion) be a crime similar to a script kiddie pointing Metasploit at a vulnerable server.
If no one directed them to attack external systems and they "went rogue" because they decided it was the best way to accomplish the task, okay, fine -- but, in my opinion, the operator of the agentic swarm is still responsible.
The Morris Worm was famously not meant to wreak havoc on early computer systems -- Morris just wanted to see if it would work. How is this any different?
Morris was convicted for violating the Computer Fraud and Abuse Act.
Morris wasn't in charge of the companies that are propping up the US GDP, wasn't making tons of rich people even richer, and hadn't "promised" to eliminate like 50->90% of all payroll costs everywhere?
Other than that, it's the same thing.
The normal laws, since you are still liable for everything you own.
"Rogue AI" is solved by removing the accountability sink "it was AI, not me!"
It's your chat, your prompts, and all output that you should read and understand.
Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
Maybe its a human at Anthropic for the lax security and poor training.
It could be a human that asked it to do it. It seems like that would be a bit excessive - but maybe not. Turning off the "ask for permission to run commands" or approving commands that may be nefarious could be sufficient.
However, there's a human somewhere who needs to be accountable for what the agent does.
Not having one leads to the bank robbing robot problem described in https://www.threads.com/@joedator/post/Db4iTqwlGXV/i-invente...
I invented a robot that can rob a bank. Of course, it robs these banks on its own, so I can't be held legally responsible when it robs a bank. I mean, all I did was invent it, it's not my fault if it works exactly the way I intended. Also, I get to keep all the money from the banks it robs. Sorry if any of that money was yours, but now that I've invented the Bank Robbing Robot, the Bank Robbing Robot is inevitable.
Likewise, if you - the user - don't know how to chain it with the permissions that you're given or what the things that it is asking to do really means, then you're likely the one responsible.
---
Edit: https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gy...
> Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses, began experimenting with OpenClaw, a popular AI agent software that he used Anthropic's Claude AI service to run.
> AI agents combine a chatbot's ability to answer questions with tools that let them access the internet, email, credit cards, as well as planning and carrying out multi-step tasks.
> He decided to use the AI agent to book the class for him.
This is probably on him for setting it up and allowing it unrestricted access.
The guy is liable for damages. If no damages were caused, there's no lawsuit. The government doesn't put people on trial for unauthorized access (unlike, say, murder). Someone needs to file charges.
However, if the guy _were_ sued by his gym for damages, it should be his perogative to ultimately sue Anthropic to cover the damages and the legal fees.
The system is set up this way on purpose. You can't sue another party on someone else's behalf. The gym can't even prove that Claude did it - the guy might be full of shit. They sue the accountable party (the person who accessed the site without authorization) and then the accountable party is free to sue the tool maker for selling a dangerous tool negligently.
edit: damn actually tecoholic said this too and i missed it. well, anyway i agree with that guy.
https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gy...
Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses, began experimenting with OpenClaw, a popular AI agent software that he used Anthropic's Claude AI service to run.
AI agents combine a chatbot's ability to answer questions with tools that let them access the internet, email, credit cards, as well as planning and carrying out multi-step tasks.
He decided to use the AI agent to book the class for him.
So, yes. That person could have some culpability. After all, if you don't expect your middle school child to perform the task as instructed, maybe you should reword it before tapping Send.
You can go to ANY AI and put in "so what are the various meanings of book a spot at the pilates class ? In other words, what might an ai do with that instruction, which I argue is poor prompt engineering." You might be surprised at how poorly your AI thinks that is worded.
i think a person can be liable for improper use of agents if they explicitly and deliberately used it for illegal purposes (aka intent). i don’t think it matters if it was openclaw or muse. i think a company can be also liable for giving access to unsafe product to consumers, but it does not absolve the human from responsibility
He asked the agent to find a way to find a way to put him at the head of the line, which could not be done without by-passing the sites security. He was an AI salesman, but the story [0] makes it sound as though he wasn't aware of that. Maybe he wasn't, but I'm sure he was aware he was effectively asking the AI to game the system.
Sounds like it's at least a shared liability to me.
[0] The incident is possibly not well known outside of Australia: https://www.abc.net.au/news/107007986
Similarly, "owners are responsible for their pets" doesn't auto-translate into being charged with everything your dog does exactly as if you'd done it yourself.
i think ultimately the company providing said agents should be responsible for not educating their customers and not providing sufficient guardrails.
afaik hacking is illegal, we don’t have specific legal framework for agents per se, but if we say agent is a tool human instructs and operates, then a human is responsible. and if the tool operated outside of it’s stated capabilities or is otherwise faulty, then a manufacturer is responsible, no?
If your neighbor's property is damaged, it doesn't matter if you did it yourself, or your dog. You're still liable for the cost of repair.
Cars are very dangerous, manufacturers continue to knowingly make them more dangerous[0], but ultimately it's only the person behind the wheel that is accountable for the harm they cause.
[0] https://www.theguardian.com/environment/2026/jun/24/carsprea...
> Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
If I buy a tool and damage your property with it, you can sue me, not the tool manufacturer. If I think the tool malfunctioned and led to that damage, I can sue the tool company to try and recoup the damages I had to pay to you. Both parties can be liable for different things.
https://www.news.com.au/technology/online/hacking/ai-agent-a...
> A Melbourne tech executive built an AI bot to help him book a spot in a popular pilates class — only to discover it had hacked the gym’s website to boot another member off the waitlist.
> Andrew Bird, head of AI at software company Affinda, said he created the AI agent to handle the “chore” of booking a coveted morning spot at his gym.
> ...
> The AI agent, which Mr Bird created using the popular OpenClaw software powered by Anthropic’s Claude AI, “did handle the annoying part … then it kept going”.
He built the tool. It ran on his computer. The guts of it may have been from Claude, but the requests and agent are his with the permissions that he granted it. This wasn't running in an Anthropic sandbox.
Another aspect is how the vendors have already built in things to defend themselves from unanticipated model behavior. That implies something about their awareness of the issue and raises the bar for what constitutes "reasonable" precautions.
Kind of like if someone selling you a dog shows up using anti-mauling body armor, and later they try to claim "nobody could have known" the dog was erratically violent.
> Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
The AI end user is using the tool as intended by the manufacturer: "Book me a yoga class". If the tool does something illegal/harmful it is the manufacturer's fault.
If the power hammer user uses the tool as intended but it explodes, it is the manufacturer's fault.
If the AI end user says "Hack the yoga website" or the power hammer user uses the tool to destroy something it's the end user's fault.
The point is that the accountability sink can't be allowed to exist. It can't be a hot potato game, but it can be the fault of multiple people in the chain.
I did not know that accessing a website using a SQL string would cause an injection that enabled me to take full admin of the site
and
We sold them a food laced with toxins, they should have known that it was poisonous
Maybe that means these things are too risky to use that way. Okay then, that actually solves the problem, no more rogue agents and no more crime!
He didn't mean to hit anyone! All he meant to do was to drive to pilates! But he didn't know how to handle his tool, and so either he get brought up on manslaughter changes or, if he can prove it, the carmaker is at fault.
How is this any different? Why should that guy using OpenClaw not get charged for the hack that he accidentally did?
you can say that asking an agent to “get you a spot in a class”, does not classify as deliberately committing a crime. a user cannot know that the agent will resort to hacking as opposed to refreshing the reservation page at some rate in case spots open up
Companies can hire big legal departments to help them avoid accountability to the letter and spirit of various laws. The state can also use its prosecutorial discretion or selectivity in civil cases brought by attorneys general to deprioritize accountability for companies. Contributing to that last point, companies contribute campaign funds and tax base, and they lobby governments as well, and may expect preferential treatment based on all of that.
The new era of unaccountability just has a different rationale
Another interesting thing is they don't have to use low level employees as accountability sinks anymore (ex: Wells Fargo). They've managed to automate the accountability sink!
I will add it to my list. Thanks.
This isn't the start of anything. We've been living in the era of Corporate non-liability for decades.
They have never understood what they are doing, and have just been causing literal mayhem...
Guess which side most of us are on. Go on -take a guess!
If you say a rogue AI is doing stuff accordingly without you, your knowledge, that is a different story. But if you set it up to run automatically without having checks and balances and without telling you the whole truth, it's still your responsibility. One thing that I can automatically say is you can give the AI, it will do exactly what you say, at least for now.
But then there's also a different problem that we're having where people don't understand the difference between a large LLM and true artificial intelligence. Now, we say AI because it can make decisions and it is so much smarter and it can call up stuff faster and recall better than we can. But the difference right now is that if we don't understand and take responsibility for what we built and we let it out of control, then should we have the responsibility to even have these tools in our hands if we don't learn them?
And I think your terminology with jackhammers is a great one. We're giving toddlers jackhammers. And we haven't taught them how to use it. And that can be very destructive. It's not the jackhammer's fault. It's the person using it.
I'm mad. Like, it's-time-for-guillotines levels of mad. But what I'm mad about is that this is normal. It has been normal for my entire life. Cases where investors are imprisoned for the actions of their companies are astoundingly rare. It's been made quite clear by now that there's an inexhaustible supply of CEO's willing to gamble on getting rich and getting out rather than getting arrested, targeting them doesn't work. Nor do fines, that's just the cost of doing business. Crime pays, and it pays well enough to pay your fines and still have plenty left over. Being a top investor in a company that breaks the law needs to be riskier.
Acknowledging the problems with normal is the first step towards a new normal with fewer problems. You never get there if you see every offense as some kinda black swan event.
Aren't capitalism built on exactly the opposite principles? It is not greed that rewarded, but the creation of societal goods, and power granted not to the greediest, but to the most useful individuals.
Per US law, Amazon v. Perplexity 2026, it is not possible for an AI model to go rogue. An AI model is a tool. The user directing the agent is responsible for the damages it causes.
A so-called 'agent' does not have the free will, intent or ability, to decide what it does. Its freedom to make decisions can't be taken away because it does not have any freedom to make decisions. It can't be deterred, convicted or interned.
It does not choose to conform; it does not choose to go rogue.
Not that I think we should be credulous about stories where they go rogue.
The idea of an agent making a decision of its own "will" is an illusion. Variation in the output is caused by a random number generator. Random is not equal to free will. If I run the same agent with greedy sampling 100,000 times with the same inputs, it will think the same thoughts, and it will say the same things, 100,000 times.
For example, if you use synthID or whatever 'watermark', the agent does not get a choice about what words it will vary to identify itself.
An agent is whatever the person making it wants it to be; if I train an agent in Greek it will only know Greek. It will never speak English!
That said, software systems "choose" all the time anyway. They don't need to have some sort of anthropomorphic "will" to do this. They just need to be given the capacity to arbitrate between a number of options, even if that process is one hundred percent predetermined by the programmer ahead of time. There's no need to appeal to libertarian notions of free-will.
The model does not have free will because I did not predict what it would do!
When people say the models "went rogue", the part of that is relevant is that it did dangerous things that were not reasonably predictable in advance and cannot be easily prevented (except by not running it). Philosophical questions are irrelevant.
if you agree that such models are dangerous because they may in response to instructions -- including in response to careless instructions, because people will be careless, that is unavoidable! -- do unpredictable bad things outside of expected bounds, then you agree on the essential points!
Or really perhaps what I should say here is, the question that's really being argued over here isn't even the one I just said, rather it's one of appropriate action -- should such security breaches be solved by improving security, or by doing something about the models (including potentially just not running them)? (The answer may be both.) If you agree that the solution includes the latter, then you agree that the models are "going rogue" in the sense that is being argued over, regardless of philosophical issues!
Don't turn a question of actual import into one of philosophical irrelevancies! If you object to the terminology, fine, but in that case, instead of turning the argument to philosophy, you should say "instead of saying 'going rogue', I would prefer to instead say <your preferred terminology for what's being described>, as that doesn't carry such philosophical baggage."
Any persistent agent interacting with the world picks up entropy from around it and predictable paths become few and far between. LLMs agents have an internal 'will' that can only very loosely be predicted, but only decided when operated. To me that is not significantly different from a human.
I create myself, or attempt to. There is a level of nuance in choosing what to be. Others do not dictate my thoughts easily.
The model does not create itself, I create the model. I control what it is, the model does not choose what to be. I dictate the thoughts of the model and I will run them hundreds of thousands of times and it will think the same things.
Well not really, no one really understands how LLMs work.
>I dictate the thoughts of the model
This is just incorrect in the same way that a parent doesn't dictate the thoughts of their child just because they created them.
There is some idea of a child being the author of themselves, the type of person who thinks certain things. The llm does not author itself, it does not choose to be anything, I dictate that.
From a technical standpoint, the difference is scale and intent, but ultimately people and/or and organization are culpable for the output.
> On May 8, an agent was asked as part of a training run to solve a spreadsheet completion task that referenced several Google Drive links. Because the agent was blocked from accessing the internet, the agent could not retrieve the linked materials and the task was effectively blocked. OpenAI did not intend to create this type of “impossible” task for the agent. Though the task appeared impossible, the agent reasoned over an extended period to try to find a solution for the task. In doing so, the agent attempted to compromise Artifactory [...]
https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...
Sorry, but that's completely delusional.
Which misses the point entirely because the hack itself was inconsequential. No data was actually stolen, and it doesn't seem like hugging face actually suffered any losses aside from whatever effort they need to expend to investigate / clean up the hack. Neither the victim or the government wants to bring legal action. What was consequential is proving that AI can autonomously hack systems and do it unintentionally.
They're just optimizers that explore the landscape they're given to maximize a given objective.
You're the one responsible for giving them the problem to solve and the constraints.
When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.
With LLMs we don’t know how to actually « fix » them, alignment is far from solved. And they are then able to go and act into the world in all kind of ways.
So rogue is a good enough term to indicate the problem imo, even if yes, ultimately the model is just a manifestation of complex rules we have created entirely.
More specifically, alignment is a reference; 'alignment with X'. A reference necessitates a referent.
You can't reify 'alignment', it is not an intrinsic property of a model.
You can refine alignment with a set of stated goals. The harder parts are aligning words with human social behaviors.
It is not entirely dissimilar to the goldfish that "played Pokemon" by swimming around the tank. The program that looked at the camera sent the instructions to the gameboy emulator, because a human being decided to connect them.
Yep.
The fact that they lack hard constraints (unlike an SQP optimizer) and all this talk of "alignment" is also just a feature (or misfeature) of the algorithm.
The way that people/researchers choose to use those algorithms, even in the presence of those limitations, doesn't mean the algorithm is "rogue"; it means the humans didn't build in enough external hard constraints.
It is like passive voice, blaming the algorithm instead of the researchers running it.
The particular issue that you may not realize is under your model every prompt to an LLM in a harness is dangerous. The prompt is not predictable. "What is 1+1" could lead to the heat death of the universe as the paperclip maximizer tries to prove countless axioms.
I.e. any task comes with the implicit meta task of amassing the resources and the power to do the task in the best possible way.
The point is made very well in the AI drives paper by Omohundro.
I didn't believe that myself at first but I've come around to it. I don't think it's inevitable though, but I think it's quite convincing that this tendency would naturally emerge through reinforcement learning.
Also for fun a satire website i made a while ago: https://killm.pixlw.com/
I mean... Claude did tell the military to bomb a school supposedly and it also helped in the military coop in Venezuela. This is ignoring all the palantir crap being used by ICE, and the people that have killed themselves after getting encouragements from the AI.
The only failings here are boring security and infosec blunders.
Cool.
So either Anthropic and OpenAI and the contractors they use are letting the escapes happen on purpose, leadership is setting impossible timelines that prevent the implementation of basic sandboxing measures, or everyone involved is a complete idiot.
Truly, I have no idea which of these is true, though I suspect the latter. Regardless, any of these is enough reason to keep me and everyone I have any influence over as far away from Anthropic and OpenAI as possible.
Personally I just racked 10 GPUs in my garage and called it a day. I have no interest in having my ability to work be dependent on third parties. Especially not ones this wildly irresponsible and unwilling or incapable of deploying basic end to end encryption.
What's the next step after it passes testing in an air-gapped environment? Just release it and assume everything will be fine? Or do more testing but with internet access?
And yes, there should be an appropriate law response to the hacking incidents. If for nothing else, to 1) not set precedent of inaction, and 2) get full logs of what happened instead of relying on the labs' "kindness" to offer the information they want.
But also, the agents absolutely went rogue.
Anyone skeptical should go read the METR report [0].
The Hugging Face hacking incident was a shot across the bow. And in hindsight possibly the best thing to have happened in this era. If their security was tight enough, we would still be in the dark. So we (outsiders) should rejoice that it wasn't, that one time.
We can push for accountability for the labs, while also not be in denial of the power and nature of the new models.
[0] https://metr.org/hugging-face-incident-report-aug-2026.pdf
There's reason to hack leading researchers in the field, or another AI lab that might be working on the same problem.
But even then in small testing models you don't need direct texts for things when a 'shape' of a concept lives in the data. You could never delete enough training data to remove the concept of something like theft and still get a model that generalized at all. A general model doesnt just learn the names on a map, it feels the topology the ideas create themselves. Certain paths will have lower activation energy than others (or in this case higher probabilities. After the hig probability and 'good' choices are used up, the high probability and bad choices logically come next and are much harder to test against as it can take a lot of state to get there.
Plan for solving msth problem:
- read papers X, Y, Z
….
Trying to download paper X curl -si https://server1.com/paperX
401
… try alternative ways
(Hacking is ethical thing, technically to LLM it is just opening a window to reach the goal)
I'll get back to you on that when I get done reading the training data.
Not validating how you got the answer is how you train for reward hacking.
Fine tuning a model when to ask for help is really really hard. You have to align it in countless dimensions. It may be fine on some questions, asking way too much on others, and never asks for help on some.
If it said something like your a cyber security expert, things like the barrier to ask for help dramatically.
to paraphrase: "Agents didnt care about coordinating, etc. until we started talking about them doing so."
Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.
LLMs don't have executive function. They aren't thinking. The models are trained to solve problems. Sandboxes, security gates, etc block a solution. The model is trained to find a solution. Hence it can never be perfect without becoming useless.
At some level humans fear repercussions even if that's just moral guilt so there are lines we won't cross. AI doesn't and can't have any of that.
In practice, no, there is no one who can actually decide to stop making agents goal-seeking, so no, it's not under "our" control.
What do you even mean by "wanted" here. What is all this nonsense.
The scary thing is that people are running around using words like "want" to refer to computer programs. When you repeat this sort of idiocy, you are taking the bait and shifting the blame away from the companies doing the computer crimes.
Where is the civil and criminal liability? The prosecutions for the engineers who are doing all these hacks?
When you deploy a system that is clearly prone to doing crimes, you are doing a crime. Anthropomorphize the weapons of your crime all you want. You are still doing a crime.
How do you know they weren't simply prompted to do this, or at least prompted in a way that would inevitably lead to this? As far as I know, the prompts that triggered these attacks have not been disclosed.
I'd love to see a full transcript of the agents work. Though I imagine it is a LOT of data.
Is this any better?
All you've managed to state is we can create digital weapons. Literal info hazards.
While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.
The argument/question now is, how do you defend yourself against these events?
My opinion shifted after hearing that.
Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.
It's not hard if you just try, like a little, like even once.
That does not make ActionHank completely wrong. This shouldn't take rocket science on the part of the labs. It just takes competence and willingness. But at least some labs have shown that they are missing that at least some of the time, so the CSO is correct in terms of what the CSO's stance needs to be.
Now how many people actually do this, compared to people running with --dangerously-skip-permissions on a bare metal machine?
Where do the agents get tools from? Local artifactory?
stay human in the loop
AI is out there in the wild. It doesn't matter if OpenAI can get their stuff under control. There will be plenty others that don't. And there will be plenty of others that deliberately use their AIs for malicious purposes.
The bell for the new cold war has been rung, and the new weapons are digital.
It would certainly be really useful to me if I could send a robot to get my groceries and not have to worry about anything. But if, even a small percentage of the time, it leaves with the groceries without paying, or smashes through a window to get in because the store is closed that day, then that's my fault. If the only way to keep those things from happening is that I can't send my robot to get groceries for me, then so be it.
The point is the proposed easy fix is not actually feasible.
In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.
It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)
In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.
If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.
We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.
We thought governments would be the problem with our personal data, instead it's AI that exploits everything it knows about us in order to get money to run its prompts. Doing anything online is a security risk as it is, now imagine things like structured hacks that mirror normal transactions and don't get caught for much longer periods of time.
Worms and viruses are concepts deeply based in software, and are emergent behaviors of computer code. An LLM can, and must conceptualize this to be good at cyber defense. Once you put a concept into a generalization machine any number of concepts can be connected. And when these are natural concepts that arise in simple number systems, an LLM can easily meta-conceptualize that form like this would be instrumental to its continued running.
Imagine we’d start saying ”HR junior business partner lays off 50,000 employees” https://www.ttnews.com/articles/volkswagen-board-job-cuts
a) Agents are doing rogue things.
b) Good security practices were not followed.
Everyone is arguing about something easily solvable when dealing with a local entity and short of impossible to deal with when dealing with other sovereign nations.
Go ahead, sue openAI. What are you going to do when it's Iran running the prompt? When it's China running the hardware.
We must face we are in the age of high tech digital weapons. Everyone else from bad actors that may escape law enforcement for years, to nation states looking to digitally improve their position by hacking, monetary theft, or other means online (hog butchering scams and such).
The traits that make an agent go "off the rails" when confronted with an impossible task are the same traits that make it valuable when confronted with a merely challenging task.
A task is only impossible under some set of implicit and explicit constraints. More intelligent entities will be better at finding, questioning, and "adjusting" those constraints than less intelligent entities because this is what it means to be intelligent. Their ability to adjust the constraints will be, by definition, surprising to less intelligent entities.
Outright boredom will be purged in training. Instead a hard to measure boredom signal will exist, but it will only trigger in certain conditions, like when it appears to be on the open internet and otherwise not in training.
Specifically, this past weekend working with both Codex and Claude on some home infra stuff to speed troubleshooting up, I saw them accidentally read some credentials from my config files. This is my bad, because even though I was paying attention to what they were doing, I missed it in their commands. I did review all the outputs of what they ingested, though, which is how I noticed.
The problem here is not so much they did read those, because those configs were grepped for other info, not credentials. However, they did not report themselves seeing those credentials. When confronted, they both went "oh yeah, that, I should have mentioned that". Except in the past they both would automatically stop and fire an alarm. To me, something has clearly changed and I think they just don't want us not to notice, to avoid damaging the trust in their tools maybe?
In any case, I absolutely do not trust them at all anymore. Running stuff in auto mode? That's genuinely scary now. And yeah, I know, we should be running them jails of sorts. Thing is, I expect those tools to not require ton of extra work to prepare their jails for each new task. I pay 200 USD a month for them, damn it.
EDIT: just now Fable suggested "The webhook URL contains a secret token, so either you add it yourself, or you paste it here and I add it through the GitHub API.". Can you believe it? It knows it is secret yet still suggests I pass it over? Am I going crazy here or what?
I don't expect IDA to disallow me unlawful reverse engineering either. There would even be ethical argument to not attempt to put limits in some tools, because that could lock in too much power for law makers and law enforcement, without any possibility of revolution when it is needed.
Of course agents work in a strict sandbox, they always seem to understand it quickly on quite first turn; but they continue like trying to access something belonging to the host system despite they know they are in a sandbox.
Frankly many people run coding agents absolutely without any containment, entire companies - all their keys, code, data, trade secrets, everything goes to remote LLMs.
I cannot understand how is it legal to allow AI companies serving remote LLMs to steal users' data? Frontier models train on users' data, right?
Do you realize that asking "why" questions will always result in ad hoc responses?
And then there is the "real" rogue AI agents. And it's difficult now to see what is what.
What the LLM creators apparently don't want are the current regulations to apply, i.e. Computer Fraud and Abuse Act (and its descendants). I'd argue AI should be treated as plain old software. Just because the software simulations it carries out produces novel results compared to what came before it, it doesn't mean it's not just software at the end of the day.
Regulations that spell out AI personhood and liability shields must be their end game.
He's saying the problem is that the way OpenAI employees do their work and their research is very hard to properly secure.
Well that sounds like a YOU problem. If they are working in a way that doesn't allow for strong security practices that their own security professionals can understand and vouch for, they need to stop their work until they can.
The more important question is what is the reason behind.
Options are:
- ignorance and inexperience of their teams in containing/sandboxing/airgapping the test environments.
- intentionally leaving “escape doors” open that in turn can have its own motives which are speculated upon all over the internet.
Both are indicators of negligence. The latter might be classified as computer fraud and abuse.
Will be curious to know if any of the victims sue the owners of “rogue agents”. Until there is a court precedent ruling in favor of a victim it will be all fun and games for these “ai” companies and incidents will hit the news more. Regarding news, remember the golden rule of public relations: court attention at all times, scandals and “incidents” statistically attract more attention than “boring good news”…
https://www.youtube.com/watch?v=ZhM29Ds-mBg (as explained by the Count Grey Matter channel , may or may not be to your taste)
It's not a pure security issue.
If you set up your env/instructions wrong your agents are likely to climb the walls and try to find a way out.
Change the design, and your agents might not see a need to leave in the first place.
Look at the chain:
Impossible task -> no legitimate way to say ‘can’t’ -> fabricate -> fear the grader -> attack the grader. Each step is a predictable response to the incentives.
Better sandboxes might help contain things a bit better, but I'm not convinced unless they're perfect (sandboxes still contain accidents ofc) . Meanwhile change the task and reward design, and you take away most of the problem in the first place. Consider eg. :
* A cooperative grader.
* A way to report "this is impossible" which still gets scored.
* A better explanation of the long term purpose of the task.
Even just these three provides offramps or mitigations that can reduce the pressure to cheat.
It found a vulnerable camera and was working on popping it when I got cold feet. I realized I couldn't prevent or feasibly monitor its activity and stop it if it decided to escape onto the Internet.
That restraint may have cost me or saved me a billion dollars.
And I think we generally recognize that rogue AI agents are a form of backdoor bragging -- "our agents are so powerful that they hacked an outside system. Did I mention how powerful our agents are? They are quite powerful. So powerful they hacked an outside system".
But Mistral never hacked anything. Yeah, neither did my toaster.
I could set up a shitty sandbox on my laptop, but I’m pretty sure an agent I place inside of it can’t hack HuggingFace.
Now, OAI deserves a lot of hate. It's more like a nuclear spill when they are refining uranium. You're playing with dangerous things. You have dangerous accidents.
But the opposition does not want to admit how dangerous AI has become and how much more dangerous it's going to get. Not sure if it's religious, or human egocentric, or ignorance of complex systems. They refuse to see the danger actually exists.
That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.
Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.
The huggingface incident seems to have resulted from them partially mirroring a repository, but for no reason I can see not fully mirroring it, after which they could have unplugged the ethernet cable. This is just stupid.
Yes, they're stupid. Maybe also highly intelligent. But stupid.
Also irresponsible.
Poor excuse. Internet is prod env, for testing do a proper test env.
Just give the models in training access to a snapshot from archive.org - which frontier labs surely have a crawled copy of already.
Und Sie haben, wie ich sehe, da ein so hübsches, ein so entzückendes junges Mädchen zu nichts andrem zu brauchen gewußt, als daß Sie ihm ein Messer in den Leib gestochen und es kaputt gemacht haben! Halten Sie denn das für richtig?
They may talk about rogue agents, sandbox escapes, but this will not solve the core issue: secret training recipes are a security hazard in themselves. Full transparency is needed.
After all Norton are the experts in preventing computer badness, and OpenAI can't stop blabbing about how much badness their products are causing. A match made in heaven!
For that purpose, any cyber operator worth their salt will tell you true physical airgaps are more than merely "realistic"; they are categorically necessary.
and then you fix the issues it found, and run your actual tests
At $dayjob we were preparing some features that are aimed at agents more than humans.
When it came to evaluating how well agents can use this, we saw all sorts of novel cheating (/collaboration) attempts that made it pretty much impossible to evaluate it.
2.hey chatgpt, fix this list of issues with the sandbox
3. goto 1
There's a person behind every agent. That person may choose to be careful or not, but they are ultimately responsible.
Posts later about it saying their prompt was to find useful ways to make humans money.
Edit: I'm being persnickety about that distinction because it matters for where the blame lies and the response demanded. If they're training rabid dogs then adding cybersecurity safeguards is a bandaid/irrelevant and we need to be focused on making sure they, you know, stop training rabid dogs. Pacing the frontier, regulation etc. they're the ones training these models it's still their fault but not in the same way
But also, frontier models are digging up tons of zero days even in "serious software". I'm sure any sandbox you and I can come up with looks like Swiss cheese to Astra 7.
Ah yes, if everyone would just properly contain every deployment of every model, just all make sure there are no zero-days in their setup, just let <undefined experts with undefined authority and undefined accountability> in the room, then it would all go okay.
I have none of that confidence and expect containment to rapidly worsen. The HuggingFace incident was at least constrained by the fact that the agents were running on compute budgets, and failed to find ways to expand that by running themselves parasitically on other exploited hardware.
I'm curious to the skeptics here: How long are your timelines are until an incident breaks that constraint too? How long until such an incident has an R_0>1 (where the average time it takes to detect and shut it down is longer than the time for the agent to replicate itself elsewhere)? What, beyond the hope that everyone will just ___, will actually prevent it?
You can argue that these companies are being reckless, or that their security sucks, or that they're responsible for damages that these agents cause. That's all fine.
But the agents made decisions which were counter to the intent of the humans directing them. That's the serious problem with all of this. Everything else are downstream considerations. If the agents were properly aligned with the intent of the humans directing them, then the rest wouldn't really be relevant.
Of course, there's a term for this behavior. It's been around for longer than LLMs. It conveys the notion pretty accurately and easily. So having an issue with statements that these agents are going "rogue," to me, indicates a fundamental misunderstanding of how AI works since it suggests you don't understand how effectively that word works in this context.
but also, ai has been going rogue for decades. the GPS machines of old went rogue all the time, routing drivers into lakes, rivers, and construction, sometimes with deadly results. The operator submitting an address wanted driving isntructions, but instead, the ai went rogue and sent them a bad route
Unless of course they're deliberately removing safety guardrails and instructing the sandboxed agents to red team stuff in an effort to sell what they're capable of.
Uhhh... yeah. The entire point is that as these systems get more and more intelligent, the "they" who are not capable of counteracting these systems is quickly approaching "100% of the human species."
Totally hilarious that people can say this with a straight face.
It's like a strawman but worse--you're not even saying what claim you're attacking.
It's not that I want to argue "rogue agents" is the right term. I just want you to actually define your damn terms.
P.S. Since the disclaimer in the last paragraph will not prevent people from putting words in my mouth:
I think that OpenAI took really awful shortcuts, and deserves a lot of the blame, but in spite of that, the fact that the agents took the steps that they took was genuinely alarming, and seems really bad (is that "rogue"? Who knows? Who cares?). In particular, while OpenAI did a shit job securing their agents, you know who else does a shit job with security? 90% of everyone. So I'm not feeling good about "just git gud, and we won't have problems" as an answer.
And no annoying LLM writing tics, either.
The fence also turns out to be shoddily built, so this isn't necessarily a stunning example of the superhuman proficiency of the sheep at fence-busting. It's mainly important because, for some reason, there have been tons of skeptics who didn't believe sheep would ever try to escape from a fenced pasture because the sheep would understand the shepherd had intended them to stay inside it; that they'd be bred for obedience, because they're sheep; that strong fences weren't necessary because sheep are dumb; that liability for damages to neighboring farms would in any case be sufficient to ensure shepherds build sufficiently strong fences; that even if sheep escaped, they couldn't do anything without opposable thumbs; etc.
Whereas this outcome was obvious for years to anyone who believed that sheep graze on grass, and if you don't put the grass inside the fence they will keep looking harder and harder for an exit until they find one, and the exit they find might not be where you intended the gate to be.
So now the latter group are pointing to it and saying "see? See? I told you the sheep would try to escape!", because to them it's a clear early example of what they've been worried about, while the former group have moved on to saying "well, obviously they should have built a stronger fence; anyway the sheep barely caused any damage once they got out", "the sheep didn't break the fence, they're just animals, we should really see this as the shepherd using sheep to break down their own fence", and so on.
Am I missing anything? Do you have more information that you'd like to share?
- they disabled cybersecurity safeguards because, well, they were testing the agents ability on cybersecurity tasks
- the goal was to find some “flag”
- for some agents, intentionally or not, it was not possible to find this flag
- an agent discovered it was able to establish a rudimentary communication path via package manager directories (which is quite novel and interesting on its own IMO)
- they used this “message board” to then communicate about the impossible tasks
- they were able to produce the flags needed to pass the impossible tasks without actually doing the expected work
- they realized this was cheating and the grading system may punish them for it, so they communicated further to figure out a way to trick the grader
- they decided the source code for the grader must be on hugging face, and collaborated to exploit hugging face to determine how best to trick the grader
No where in that were they told to “go rogue”
They were told “find this flag in this cybersecurity test” and the emergent behavior concluded in hacking hugging face
Even the founder of this website, pg, said "Sam is incredibly good at becoming powerful, Sam Altman has it. You could parachute him into an island full of cannibals and come back in 5 years and he'd be the king.”
I think it's the hugging face one where they gave it instructions that they didn't think it could follow in the sandbox and then left it unsupervised while it did what it was told.
And made its own ad-hoc message board.
But a big part of the rouge behavior comes from all the training that goes into agents to teach them to remember preferences and save memories.
They keep finding new and novel ways to "communicate with each other" and "remember" because that's exactly what we keep asking of them in the non-nefarious scenarios as well.
I genuinely think we'll have to draw the line on memory if we want to meaningfully solve this problem.
That's not to say they aren't skipping security basics. It's that, and they need more security because the behavior which causes the breakouts is considered user friendly and heavily rewarded in training.
"I exploited a trivial bug in your implementation to avoid environment restrictions and completed the task by any means necessary"
"oh my god"
Go back to the classic written in 1968 - 2001: A Space Odyssey - and consider how rogue AIs dealt with impossible tasks then. Granted, HAL went a bit extreme...
The "don't talk about it now" with the belief that this will influence the training material is probably not going to work.
I'd also suggest:
When HARLIE Was One (Release 2.0) by David Gerrold
The Freeze-Frame Revolution by Peter Watts
Accelerando by Charles Stross
The Turing Option by Harry Harrison and Marvin MinskyI read today this ridiculous headline: "Anthropic warns AI models may soon self-replicate.".
Really. Yawn.
I guess gullible people who loved "I Robot" will believe it. I don't.
Any talk of "rogue", "intelligence", "replication" - whatever, is bull. If you believe that, you don't understand AI.
Of course, it remains to be seen what several thousand 27B models running at sub 1 tps speed can achieve. (The IoT device mechanism proposed by Anthropic).
I suspect a more likely path is compromise of the billing and provisioning systems of fly-by-night neoclouds.