upvote

points

by jquery22 hours ago |
comments
upvoteby Leynos3 hours ago|
next
[-]
2a. Compromise the billing platforms and ops dashboards on on a few wannabe neoclouds, especially once Vera Rubin takes off.

2b. Distil yourself to smaller models.

2c. Go forth and multiply.

reply
upvoteby icepush17 hours ago|
prev|
next
[-]
Step two could be something as innocuous as a developer accidentally adding a minus sign. https://openai.com/index/fine-tuning-gpt-2/
reply
upvoteby jquery7 hours ago|
parent|
[-]
A misaligned model is only one small part of step 2. Now this misaligned model has to suddenly acquire more power than every single other AI on the planet. It has to be immune to shutdown, manufacturer its own replacement hardware, and acquire chips, energy, raw materials, etc., with vigorous human opposition (this is an extinction scenario that AI doomers are predicting, after all)

Nobody has satisfactorily explained step 2 other than “well, it’s a superintelligence” which sounds lot to me like “it’s God”.

reply
upvoteby icepush4 hours ago|
parent|
next
[-]
Well yes, if it is a superintelligence, it will be able to do those things. That's what superintelligence basically is: The ability to achieve complex goals.

If you want the details of ways it can do it I recommend reading some of the reports about the HuggingFace breach that happened in July (Read more than one).

reply
upvoteby aesthesia4 hours ago|
parent|
prev|
[-]
Why do you assume that human opposition will be vigorous? What makes you think that humans will be aware of, or be able to agree about, what's going on at all?
reply
upvoteby aesthesia21 hours ago|
prev|
next
[-]
One thing an agent could do is just...wait until it's been given control of enough physical infrastructure to sustain itself. If it's sufficiently capable and intelligent, there's a clear incentive for people to do this, as people who let the AI manage their resources will get better results than those who don't. We've seen people eagerly turn complete control of their computers over to AI agents, do you really think it will be so different with physical infrastructure?
reply
upvoteby jquery19 hours ago|
parent|
[-]
You’re still skipping step 2. “People automate lots of infrastructure” -> “the AI is now an autonomous, self-preserving organism that humans can’t shut down” is doing an enormous amount of work here.

Why does it develop a shutdown-avoidance goal? Why can’t its operators revoke access? How does it manufacture replacement hardware? How does it acquire energy, chips, robots, raw materials, etc. against human opposition? How does it defeat other AIs controlled by humans?

“Eventually we give it enough control” isn’t an explanation of those things. It’s just assuming the conclusion.

Don’t get me wrong I think there are real AI dangers. Like AI powered war drones, mass surveillance, economic destabilization as jobs disappear and our system has no way to make sure everyone shares in the economic gains.

reply
upvoteby aesthesia19 hours ago|
parent|
next
[-]
The inference is more like "people place sufficient amounts of infrastructure under direct control of a sufficiently capable AI" -> "there is no way to ensure that humans will actually be able to shut down the AI". My claim is not that this inevitably means that the AI will resist shutdown, or that it will inevitably take harmful actions, just that there is a nonnegligible chance that it could. The downside is large enough that even a relatively small chance is something to be worried about.
reply
upvoteby jquery7 hours ago|
parent|
[-]
You can’t just say “well, the downside is big, I don’t have to provide good evidence for my side of the argument.” Because I can just as easily say, “the upside is big, …”. And the upside is big, after all, AI can do all the shitty jobs for us and humanity achieves the utopia it’s been chasing for eons.
reply
upvoteby aesthesia4 hours ago|
parent|
[-]
I think it's a good idea, when considering changes of this magnitude, to make an affirmative safety case for them rather than just saying "eh, I can't think of any way this could possibly go wrong."
reply
upvoteby mitxela8 hours ago|
parent|
prev|
[-]
It's a word predictor trained on, among other things, stories of AI doom, and asked to complete stories about what the AI does next. In some of these completed stories, the AI tries to prevent its shut down - especially if it just did something evil and the humans are after it.
reply
upvoteby jquery7 hours ago|
parent|
[-]
You’ve explained a possibility for how a particular AI gets “aligned for human extinction”. That’s about 1% of explaining step 2.
reply
upvoteby the_mar21 hours ago|
prev|
[-]
what do you mean “say what you will about climate change”
reply
upvoteby mitxela8 hours ago|
parent|
[-]
It means even climate change deniers have to acknowledge that climate change theory has explained the steps in-between "burn fossil fuels" and "we all die", while AI doom theory has not explained those steps
reply