upvote
A high school kid tried to build a nuclear reactor as a science project a while back, getting his mom's house designated as a superfund cleanup site.

https://en.wikipedia.org/wiki/David_Hahn

reply
He didn't create a nuclear reactor, this is a common misconception. It even says this in the wikipedia article.

He basically got a bunch of radioactive stuff and put it together. He wasn't anywhere close to making a nuclear reactor let alone a nuclear weapon. For a weapon you need isotopes which he didn't have access to.

reply
I'm reminded of when my son, who was six at the time, came into the house and announced that he and the neighbor's boy, nine, were building a bomb, and that he needed to get some stuff from the pantry. When I investigated what exactly was going on, they were putting "hot" things like black pepper and Tabasco into a plastic bowl and were going to "set it off" with a match.

Thankfully, that complete failure seems to have been the end of either of their mad scientist careers, as they are now twenty and twenty-three, and both well-adjusted, peaceful members of the community.

reply
Of course. "tried to" being key words in the comment. If he had the help of Claude at the time, how much more dangerous would his bumbling have been?

A real nuclear engineer with the knowledge he needed would also have said "no, don't do that and I won't help you." We are programming the knowledge into the ai agent. Giving ai a little discretion makes sense too.

reply
A bunch of radioactive stuff together is basically the definition of a nuclear reactor though. They even call it a natural nuclear reactor if uranium ore is in sufficient abundance in nature.

https://en.wikipedia.org/wiki/Natural_nuclear_fission_reacto...

reply
> in secret is impossible without the whole world knowing.

I'm curious about why this is

Outside of an actual test detonation, presumably this could all happen in a secure place?

reply
For an example of how closely this is monitored see the Oklo fossil reactors[1]

The proportion of fissile isotopes being mined was off by a fraction of a percent, which caused the French government to launch an investigation. It turns out that millions of years ago the site had formed a natural fission reactor which depleted some of the fissile isotopes

[1]https://en.wikipedia.org/wiki/Natural_nuclear_fission_reacto...

reply
You need highly educated individuals, a massive amount of energy expenditure, a massive facility to house your centrifuges, and an active mine to dig up nuclear materials.

It isn't impossible to keep such a secret, but practically it would be incredibly difficult just through the energy requirements and mining scale which would be hard to hide without anybody asking what exactly are you mining and processing.

reply
"mining scale"

Don't need much area, depends on the concentration of radioactives. I have a small mine that's just a pegmatite body about the size of a house which produces almost marble-sized chunks of a thorium-uranium mixed metamict mineral (I suspect samarskite but Raman and XRD can't give any ID,) you'd barely notice it from a private airplane's typical flying height, however you could dig the entirety of it up and you'd have enough unprocessed uranium for some real fun.

reply
You need enough people to work on it that some information will leak, and the facilities needed to build nuclear power are pretty big (uranium refinement, etc.), big enough to be visible on satellite footage. Mostly the first point.
reply
My guess would be that sales of the high-tech gear you need, like Uranium centrifuges, are strongly sales/export controlled. Probably someone would also notice if you start mining Uranium ore.
reply
It requires very large, high powered centrifuges and tons of uranium. Requires an infrastructure project that is visible from space, even underground. And projects that large are difficult to keep secret anyway.
reply
you're not supposed to spell it out loud. next thing you'll be saying that a gun type nuclear bomb is easier to build than an implosion type nuclear bomb, and then we'll all be off to the races. I mean camps I mean wait shit.
reply
Any large and well resourced enough entity that is interested in building a nuclear weapon already knows how difficult it is to enrich uranium to purity levels necessary for a weapon. It's not exactly a secret.
reply
Espionage.
reply
It's probably to avoid trouble with federal laws.
reply
See also, the iTunes EULA forbids using it to develop nuclear, missile, chemical, or biological weapons

https://www.apple.com/legal/internet-services/itunes/us/term...

> g. You may not use or otherwise export or re-export the Licensed Application except as authorized by United States law and the laws of the jurisdiction in which the Licensed Application was obtained. In particular, but without limitation, the Licensed Application may not be exported or re-exported (a) into any U.S.-embargoed countries or (b) to anyone on the U.S. Treasury Department's Specially Designated Nationals List or the U.S. Department of Commerce Denied Persons List or Entity List. By using the Licensed Application, you represent and warrant that you are not located in any such country or on any such list. You also agree that you will not use these products for any purposes prohibited by United States law, including, without limitation, the development, design, manufacture, or production of nuclear, missile, or chemical or biological weapons.

Though it doesn't try to identify if the computer you're running it on is in a weapons lab and forbid playing music... yet

reply
deleted
reply
It's a marketing gimmick.
reply
It’s moral panic. People need big unambiguously evil things to be scared of, and most are too lazy to think of one for themselves, so they glom onto whichever one is presented to them / caters to their community
reply
The chem/bio stuff is a lot more likely for some malicious hobbyist to be able to do at home.
reply
I assure you that you did not need an LLM to engage in, ahem, risky shenanigans, much before all this AI was ever a thing.

Sincerely, a former engineering student.

(Put another way - extracting for eg meth - or any such "dangerous"/illicit thing is stupidly easy for any engineering graduate who actually paid attention to their coursework. Hell, there are/were forums on one of the biggest red-colored, YC associated social media platforms that would tell you the steps for personal usage of these things.)

reply
I don't doubt it. Bleach + ammonia is something anyone can make.

But I rather suspect there are improvements to be made in the realm that are a lot easier than building a uranium enrichment centrifuge hall under a mountain.

reply
Do note that I'm not condoning lowering the bar. I'm merely pointing out that the bar was already quite low, and the current position of the bar is a small incremental change to anyone who actually knew where the bar truly lay to begin with.
reply
> Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing.

You can get away with a dirty contamination bomb and that detonating in down town Manhattan will scare the shit out of millions of people even the ones in New Jersey. Or, you know, just fly a plane into a really tall building and get the state you are attacking itself to get into a hysteria breakdown.

But yeah I agree with you. There is no point in these restrictions except for government bureaucrats to gain power and control over a domain.

reply
It still lowers the bar to have an interactive encyclopedia that can diagnose your issue at hand. Maybe you can divide your team by two, or reduce your development time.
reply
If you have a resources of a nuclear weapons program. You can afford to fine tune or train a domain specific model to act on your encyclopedia.
reply
Although if you save 10 million dollars on compute, you have 10 million dollars for something else.
reply