upvote
Speaking to "break AES" - this is the wrong way to look at the problem. If AI finds AES is broken, then it was always broken, and we did not understand yet how. This is why you can "discover" zerodays - they were there all along. I prefer to know about the risks and design around them, than avoid the problem entirely (leaving them open to exploitation by bad actors).
reply
I myself work in the cryptography department of a large company, so I'm not arguing the semantics of cryptanalysis, or crypto agility, or patching. I simply meant "imagine if suddenly anyone could decrypt AES ciphertexts", and that society is not ready for such sudden disruption.

It's like everyone suddenly developing telepathy. It's not dangerous per se, and we could adapt if done slowly enough, but all at once? Catastrophic.

reply
Society is never ready for anything. It always progresses reactively only in response to technological advancement.
reply
Yes, and BoppreH is arguing for the reaction right now.
reply
I'm not sure if I'm so comfortable with this dogma when it is applied to, say, our immune system.
reply
The only reason nuclear nonproliferation works is because the inputs are relatively rare and traceable. (Yes U238 is abundant but you can’t mine enough of it or buy the centrifuges to turn it into U235 without getting noticed.)

I have all the tools to develop my own ML training pipeline on my machine today. In order to choke off the inputs to “LLM proliferation,” we’d need to confiscate every computer, destroy all copies of the Internet, and throw out all books and other archived media. It’s not gonna happen.

reply
RAND & others have published work about this, and they believe frontier model training would be much easier to track than nuclear weapons were

You need a datacenter larger than a football field that draws the power of a small city to train a frontier model, and need chips that are produced by the most complicated supply chain in history (owned almost entirely by US allies like Taiwan & Europe). And that's before you consider on-chip mechanisms that seem entirely feasible. This is much easier to trace & much harder to reproduce than precision spinning containers.

Though I agree seems much harder to detect people running models that were already trained.

reply
> The only reason nuclear nonproliferation works is because the inputs are relatively rare and traceable.

There are only three companies in the world capable of mass-producing the chips used in SOTA AI training, they all use machines from a single company (ASML), and it also requires enormous power-hungry datacenters. Meanwhile, uranium can be dug up from the ground. You say that buying centrifuges will get you noticed, but why can't we do the same for SOTA ML hardware? It honestly sounds easier to control. And differently from nuclear weapons, both sides are afraid of rogue AIs.

This is a danger to our civilization, I'm sure we can figure something out.

> I have all the tools to develop my own ML training pipeline on my machine today.

Are you saying you can train the successor for Claude Opus 5.5 on your machine?

Already today you can get on a list by buying too many smoke alarms due to the americium content, or drug precursors, or a thousand harmless but suspicious products. I would be ok with a world where you likewise need a justification to buy thousands of GPUs at once.

reply
Maybe we just have very different imaginative worlds, but after seeing the kind of crazy shit that people will believe thanks to online radicalization, I'm a lot less sanguine about the possibility that people will take a shot at your list of horribles. There are a lot of people out there who seem to genuinely believe that what's at stake is the very survival of humanity and the continued existence of truth, art, beauty and all the other good things. After enough exposure to that kind of pressure-cooker radicalization, something like a repressive world government that uses violence to ensure that only the "right" people have access to dangerous things like literacy and computation could seem like a small price to pay for saving us all. Sure, it'd suck in a lot of ways, but Altman and Amodei will have left the brave revolutionaries no choice.

I think it's very unlikely something like that would succeed, but I'm increasingly sure we're going to see serious attempts to do all of the things on your list, and they could do a lot of damage even in failure.

reply
How is solving an abstract math problem related to having unlimited destructive powers? There is no relation.
reply
My post already gave three examples. If you have a problem with those and can't think of any more, I'll be happy to provide another dozen, it's not a hard exercise. Also, "unlimited" is your word, not mine.
reply
> pause development of models any bigger, enforced globally just like nuclear proliferation.

How? American couldn't enforce its brief monopoly on nuclear weapons either.

reply
No, but we have successfully kept possession of nuclear weapons to 9 countries out of hundreds for decades.
reply
Possession as in ownership, nine. Possession as in have in their territory, fifteen.

"Successfully kept possession (limited to)" .. seems a little over confident- some of these nations acquisitions and expansions occurred despite others not wanting that to occur.

An additional twenty-eight countries have "virtual" nuclear weapons, having endorsed others to use actual nuclear weapons on their behalf.

reply
John von Neumann helped Manhattan.

https://www.youtube.com/watch?v=W06g7gIfwRE

Similarly, now apply Fields medal level AI to R and D & hard deep tech problems.

One month back,

https://www.bloomberg.com/news/videos/2026-10-02/lockheed-ta...

https://www.youtube.com/watch?v=JI7trhPHx3Y&list=PLXkles7kFx...

"We all saw the headline that they were able to overcome the math problem for the that researchers had worked on for years and we have some math and physics challenges as we employ this next sensor capability on the F-35 and OpenAI is working next to us to uncover some of those math and physical challenges."

reply
Yeah, that's how my fears start. I don't want to find out what happens to the global stage if one country develops autonomous killer drone swarms or something.

The militaries might be salivating at the opportunity, but I hope that the risk of rogue AIs brings us all to the same side on pausing development of bigger AI models.

reply
We should try to stop it as hard as we can, but in full honestly, I don't think we ever could have. I think any intelligent species that doesn't go extinct is fated to reach this point from the moment it awakens...
reply
>What happens if a jackass uses the next AI to design a supervirus

simple, just wear a mask, duh. or two. three tops, for maximum efficiency.

reply
Genie is out of the bottle.
reply
A genie is out of the bottle, but it's weak enough that the consequences so far have been limited to job loss and medium-sized IT problems. The creation of better models is not inevitable, it's done by human engineers using immense hardware, and therefore can be stopped before things get worse.
reply
Go ahead and stop. Others won't.
reply
Who gets the keys to the world ending demon?

Dario amodei? Sam altman? Donald Trump?

reply
This one person isn't a jackass. Sorry for the quibble.
reply
I didn't say he was a jackass.

Also, to preempt another comment: yes, I know that he's a great mathematician. But we've had great mathematicians before, and even working together for 150 years they were not able to prove Riemann. And as models get better, the pedigree required for groundbreaking results (including dangerous ones) will naturally lower.

reply