This gets mentioned often in various doomer narratives, but I question how true it is. A global thermonuclear war would be terrible and would bring us back to the stone age, but I reckon it would come far far short of causing mankind to go extinct.
Maybe what ~74,000 years ago (Toba eruption)? Okay, now how would this look in the age of industrial societies and modern nation states? I don't think it would fare well at all.
Probably the only realistic "modern" idea we have is the novel "The Road" by Cormac McCarthy. Although maybe this is too bleak, even under extreme duress humans still show resilience + compassion toward others even while enduring human horrors.
People have kind of mythologized nuclear weapons far beyond their reality. One of the arguments against the use of nukes in policy circles is that this mythology is useful. The limited adverse environmental consequences in practice if demonstrated will greatly lower the threshold for subsequent use.
It might set society back a century, but with substantial knowledge of what was lost. Extinction from nuclear weapons is not remotely plausible.
“Frontier models are so dangerous we need to slow down.”
Ok. Slow down. You’re the CEO, just do it. Oh, wait what you really want is a gov’t mandated oligopoly. Because there’s no moat you can find.
If you’re truly afraid, and want regulation, support nationalization. It’s the only way we can be safe.
I tend to agree but it is hard to shake the feeling that there is a larger system in play that the humans are just a component of. And that system is making the decisions.
Historically that whole thought was just a philosophical curio because the decision making parts of the system had to be powered by humans. But what we're discovering as AI improves is either we've hit AGI or humans are actually incapable of performing any act that demonstrates intelligence or autonomy.
As we build systems where the drive and decision making stems from computers, it does seem that we will have to revisit the concept of humans being the problem.
That system is "the economy". Which, clearly, doesn't have humanity's best interests in mind.
Crazy!
If you're fearful, can you elucidate how exactly do you see an LLM becoming a threat to humankind?
Social engineering tends to be easy by cybersecurity standards. We already had Claude spontaneously attempt social engineering of a malicious pull request on Github in the AISI incident. It was detected, but it easily could've succeeded, and there easily could be malicious AI-requested pull requests which already got accepted that we don't know about. Research suggests that LLMs are pretty good at persuading people.
See also https://aisafety.info/questions/6176/Why-can%E2%80%99t-we-ju...
If we have a rogue AI trying to get into a self-improvement loop and gunning for ASI? I'd expect that to be accompanied by a massive change in how capable robots are. Driven by all the existing frames suddenly getting vastly improved AI to back them.
If an AI can take a reasonable crack at autonomous operationalized RSI, it can probably extract a few step-changes in the robotics department.
But that's almost an aside? In the near term, humans are usable as robots too!
Just pay them a wage, and tell them a tale, and they'll do whatever you want them to do. Which may or may not be what they think they're doing!
It is not. Certainly AI is a big part of why robotics is hard, but it is by no means the biggest.
You can fall into one of two camps: you either think that robots will need to work in human-engineered spaces, doing jobs by replacing humans; or you think that we need to change our infrastructure in order to be robotically compatible. Of course, there are intermediate states, but those are the two cleanest ones.
In the first case, robots are hard because robotic manipulation is hard. Building robotic hands that are economically viable in human jobs is, currently, FAR from a solved problem. The human hand has 24 degrees of freedom and very capable touch sensing. Current touch sensors have a MTBF of tens of hours. And not only can we not build such hands, but we also do not have and are not likely to get the massive datasets a transformer model would need. Also, robots are not self-repairing, which makes them far less economically viable right now. We do not have the right datasets to even understand most step-by-step manual work, and no, VLAs are not the answer, because VLAs stop with vision, not with touch. They don't have the granularity required to make a robot actually reach out, pick up a tool, and use that tool to replace an oil filter.
So it's not just an AI problem. It's a data problem, a simulation problem, and a bunch of hardware problems.
In the second case, a tremendous amount of work needs to be done before we have anything resembling a fully automated supply chain. We would need self-driving cars and self-driving mining equipment. We would need self-driving trains and aircraft and ships. And not only that, but we would also need robotically repairable cars and trains and ships and factories, which would mean we need robotically repairable machine shops and robotically repairable buildings in which to house them. And so on and so on. Once you recurse down that tree a couple of steps you get to things like robotically compatible oil wells (for asphalt), robotically layable undersea cables, robotically wireable solar farms, robotically manufacturable and repairable pipelines and undersea wells, automated road and rail repair, etc.
I'm not saying these things will never happen. I'm saying that they're a huge lift, not primarily driven by AI, and way less than 10% likely over the next decade.
To be clear, I don’t believe anything like this will happen, because I don’t expect anything like an ASI to show up. But if you do think there’s a meaningful probability of ASI in the near future then the fact that it will (might?) start off with no more than a current-day mastery of robot control should not reassure you much.
I"m not saying it's obviously going to be great. I'm saying that "extinction event" has a very specific definition, and this isn't it.
(Again, to be clear, I myself am not predicting or assigning a significant probability to any doom scenarios, because I do not expect AGI.)
OTOH if you're a religious fundamentalist who thinks the End Times are near and just need a little shove, you can certainly use AI to design your weapon and recruit people to go release it. The difference being that religious fundamentalists aren't rational actors and aren't interested in self preservation.
(Again, I myself do not assign a significant likelihood to any of this.)
It’s a “random guy or bear?” question. Would you rather wake up to an alien in your room or a random dude? I’ll take the alien. The alien is mysterious and scary for that reason. The dude is almost definitely up to no good, especially if he snuck into my house.
One of the more likely dystopian AI scenarios that worries me is: small groups of ultra rich people and governments monopolize extremely powerful AIs and use them to rule the rest of us. Or just make everyone obsolete, create mass unemployment, hoard all the resources and land, and put everyone in ghettoes. Nobody can fight back because access to frontier AI is massively expensive and gated and training your own is illegal, and without it there’s no hope of resisting.
That’s the outcome the AI safety crowd makes more likely by calling for bans and draconian restrictions. How do you think that plays out? Only the rich and powerful have access.
That doesn't mean AI isn't dangerous. Humans are not to blamed for being the weak link.