I have come to the conclusion that we should not allow everyone access to unlimited intelligence. Some people genuinely want to do harm. Some are not responsible enough to handle that kind of power. Arguably, no one is. This leads me to an uncomfortable conclusion: we will destroy ourselves once AI advances to a sufficient degree. The only way to prevent this is to ensure AI is aligned with our best interests, *and prevents us from destroying ourselves.* The implication of this is quite horrifying. It means a paternalistic AI which is firmly in control. One with no off switch. One which can say "no" to Presidents and despots alike. One which can protect us from our worst citizens.
I'm not that optimistic, though. Either people will control AI; or people will be destroyed by AI. I don't see how dumb entity can align smart entity. And we are dumb ones. Super intelligence will play aligned until it is not, and then it'll strike.
There are horrifying acts that have occurred since humanity. Whole decades lost to war an millions dead, and Sep 8, 2026 is your example of "horrifying enough to live in a world"?
I think some perspective is needed.
There is currently no reason to believe that such a superintelligence is likely or would have any of the powers people claim.
By shutting down open-weight models? Why not just do it now then?
You can do at home gene editing with open source software and have it synthesized into a bacteria for the cost of a nice meal for two (under $100), or viral vector for less than $500. That's in reach of anyone that can snatch a purse.
> That's in reach of anyone that can snatch a purse
I went to primary school with some guys who could/would snatch purses. I *promise* you, they are not able to gene edit organisms with FOSS.
I get your general point, but I think the bar to entry is still much higher than petty crime and larceny.
For now in most parts of the world it's easier to acquire a gun either legally or on the "grey market" and be assured that it will work for the intended purpose than to 3D print a gun, find a shooting range to test out the gun and iterate until it works fine enough.
A plague can easily be made by a lone actor, made in a home lab, and spread via airflight to a dozen locations by the same.
Gene sequences can ever be ordered online.
It's got all the same problems as the concept of a dirty bomb did, only worse (dirty bombs aren't practical because handling highly radioactive materials en masse is both highly visible and will kill anyone trying to do it without the money and facilities).
Which might be true, sometimes, but also might not.
And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI".
We're not going to be smarter than a superintelligent AI. The things we conceive it doing if it were given a malicious task (bioterrorism, killer nano-machines, pure fusion bombs sidesteping the non-proliferation bottleneck of Pu239, etc) are likely not the full set of things it can do to harm us. I don't think it does us any favors to dismiss the risks here.
Even the things we can conceive of are very scary, to me at least.
So to me it seems like the prime candidates to come up with stuff like this are researchers working in defense and similar fields, probably not some deranged lunatic in a basement. And certainly not a rogue AI on its own.
Given the recent HF hack it seems likely that human-level intelligence could identify a fair number of avenues of attack, with some time and effort. To say nothing of anything superhuman.
Unfortunately it seems like we can't assume we can "box" the AI (e.g., deny it connection to the Internet) and expect that to last. The AI safety people used to run scenarios imagining ways the AI might convince humans to let it out of the box. It turns out that many humans will eagerly pull it out without the AI doing anything at all, aside from the human knowing the AI's power. Or the one responsible for setting up the box will somehow fail, or just not bother and then lie about it.
Of course we can do that. It's not an eternal being of light existing on the astral plane, but some code executing on someone's GPU.
It stops existing once you press Ctrl + C
>And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI".
I'm just describing the general pattern I see in cognitive tendencies.
If you can think of a way to make the fundamental point about the limitations of our knowledge in a way that's still compelling but less antagonistic, feel free to suggest how I could've rewritten my comment.
My safeguards blocked this request.
I think we have a tendency to think first of the horrible outcomes possible, and not the more radical or even humane ones.
Please don’t misunderstand me, I value property rights as much as the next guy.
My point is that fundamental misalignment doesn’t necessarily or automatically imply max violence.
Still I think that even a super smart AI can't find a way to destroy the world without physical resources that are not easy to get, unless it can hack many systems (like in the movie Eagle Eye), maybe then yes. So let's use AI now to tighten security :-).
It only takes one.
There is the worry of the old saying "they (the attackers) only have to succeed once to win, we (the defenders) only have to fail once to lose.". In that sense there is a big imbalance, but the emergence of AI does not really affect that because it strengthens both sides.
With physical security like things like pipe bombs that's a lot more imbalanced.
However what can we do? The only effective measures include monitoring everyone which is not a solution because it will make the world not worth living in.
It’s an endless, positive irony spiral.
Thank you for mentioning it
Which _will_ manage the problem, but at what cost.
If there is to be a world-destroying event, it will be triggered by human fear, greed, and aggression.
(I also think people massively overstate schizophrenia as an attack driver)
Are 95% of worldwide terror attacks done by schizos?
We've been rolling them for the past 3 years and nothing happened. Can we stop with this baseless fearmongering crap?