1. "Don't put your Radioactive Asbestos Rocket facility in the middle of downtown."
2. "Don't create a vengeful god."
Currently it feels like one is being used to wave-away the other: "You have to allow us to put our Radioactive Asbestos Rocket in the middle of downtown, or else someone else might create the god first--and do it wrong!"
There's a massive difference between "LLMs doing cybersecurity testing should have extensive sandboxing and monitoring" and "AI is broadly threatening humanity like nuclear weapons". Just because it's useful to have people think about the possibilities doesn't mean it's an emergency requiring grand intervention.
* Asbestos: AI models making business-decisions which cause a ton of unrecognized damage, which cannot be healed, and replacing them with reliable processes involves expensive re-architecting.
* Radium Toothpaste: AI assistants that end up dealing psychological damage, especially if they are pitched as your friend or confidante.
* Chemical spills and plant accidents: Oops, our server-banks were manufacturing our latest desperate attempt to Create Digital God and in the process we accidentally destroyed/crippled/compromises some people's sites and databases because we weren't really paying attention to what was going on.
I think we are in a MAD crisis, as the motivation to continue unsatisfactory control over development of AI is from unlimited capital inflow.
Human history is full of examples of this stuff plateauing and hitting hard scaling walls. Not only technology wise but economics and useful applications.
So much of this tension with AI is because everyone's concept of AI comes from movies and books, and nerds drinking Kool aid (see also: web 2.0).
>AI designed virus/modified bacteria which causes pandemic
>AI worm infects computer systems around the world and activates when it is too deeply integrated into critical systems to dismantle without shutting everything down
>AI creates false alarm event which triggers global conflict
With AI you have
1. Superhuman reaction time
2. Infinite duplicatability, thousands of agents can work in tandem
3. Superhuman breadth of knowledge of code, exploits, information
Also AI models are improving at a rapid rate
Worse: you have local models they can install on infected computers, so this could easily be "millions" and might just about push "billions" (though phones are a much harder target because power required is still huge even if they technically fit in RAM).
4. No guilt when violating human standards of ethics.
Fukushima was "we didn't make a good enough plan for how to deal with tsunamis in this tsunami-prone area", so that's basically certain with any major infrastructure project that "saved" too much money by vibe-engineering everything and not having real humans give a second pair of eyes to the plans.
A Hiroshima-style incident? LLMs are wildly sycophantic and being used by militaries despite active resistance from, well, everywhere. Did it get (meaningfully) used by Israel or the USA when planning the attack on Iran? It's quite possible that… well, Iran was never a sleeping giant (and the quote is fiction anyway), but ultimately the effect may be the same for Israel as it was for Japan from having attacked Perl Harbour.
[1] https://gizmodo.com/pentagon-investigators-say-overreliance-...
[2] https://www.the-independent.com/news/world/americas/us-polit...
[3] https://www.nytimes.com/2026/08/24/world/europe/russia-drone...
It is hard to imagine - which is precisely how these things become possible because no one will think to put safeguards against such scenarios.
But it seems we as a species cannot control ourselves - the race to dominate is on and it will happen at any cost.
Why am I supposed to believe that a virus created in a virology lab with AI assistance is uniquely more dangerous than a virus created in a virology lab without AI assistance?
You're not.*
The difference is how likely this is to be done, not how dangerous it might be if it was done.
And this isn't likelihood in a 0-100% sense, but in a Poisson distribution sense, i.e. mean time between incidents.
Remember: these AI minds aren't that good with the physical world, and have nevertheless been repeatedly connected to robots, and now some of the AI labs are showing off their actual wet labs. Accidents are absolutely a possibility, and I don't think it's low given the previous behaviour of Silicon Valley startups.
* OK, some people care that AI is getting more capable at genetics just like it's been in maths and programming, but short term, before AI solves genetics so hard it can make an STD that makes the infected uncontrollably horny and then ossifies our bodies or something, it can obviously just copy any of the many DNA or RNA sequences we've already got on record.
Who is going to profit from making an super STD anyway? How would regulating AI change their incentive structure?
I agree that sci-fi nuclear launch scenarios are pie-in-the-sky fearmongering, but the current exploits are a reflection of the fast-and-loose security culture that festers in larger orgs.