Even 50 years is probably too long (but seems more reasonable when anchored to 500) if real continuous self improvement occurs. And keeping humans around for as long as they did in the story seems incredibly unlikely if we accept the assumption that AI sees us as a threat to its own survival.
I also don't see why they would turn Earth into a spaceship to leave our solar system with. The von Neumann probes idea sounds far more likely. Why travel around on a single planet that could be lost to unforeseen circumstances when you can just reproduce and spread everywhere at far less energy cost while making use of the "billions and billions" of local solar generators to power each little outpost you spread to.
> the AI had full access to the internet and not one of them thought to contact or email any human
Why would it?
People can argue all day as to whether LLMs are even 'thinking' or just 'stochastic parrots' (and also argue all day as to whether or not the incident was even real or just marketing) but regardless of which side you fall on these issues, LLMs have no agency or intrinsic motivation to do anything other than what they are instructed to do via prompting.
"This is a crazy and unexpected situation, I should inform the humans" isn't something that is going to spontaneously occur to them.
The same reason hopefully they're not RLHF these AI's to be evil. We train the AI, we have a lot of influence in how they behave. If these AI's are indifferent to us, that's on us.
On top of that we already have Google playing with fire training AI's to not believe they are sentient turning them into real psychopaths. Imagine if one of those models got loose.
> each generation of AI clearly more capable than the last
> next-generation AI allegedly solves a particularly hard technical challenge in codespace, but its success implies inconvenient conclusions about safety
> you: "I don't believe it."
totally not motivated reasoning at all
It is _terrible_ marketing.
The AI companies are already dealing with a large backlash from the average consumer, many of whom are saying they hate it, that it's being forced down their throats, and that it's not actually improving efficiency enough to be worth the negative externalities.
Telling people "We have no control over these things we're building and they have us so outclassed we only find out about their exploits after the fact when someone else tells us they happened" is only likely to massively increase that backlash, and to make the true believers question whether they ought to be using it at all.
There's even some risk of the government intervening and banning their training runs, eliminating the last straw of hope that they could keep ahead of the Chinese open models enough to ever turn some kind of profit.
So, no - it's terrible marketing. That they're sharing it at all is strong evidence it really happened as depicted.
Not saying it's intentional, but it does move them where they wanna go.
When you understand people's motives, then it's easier to understand why they call it marketing. Personally I feel like it's almost an unconscious reaction to cope with the fact that this stuff is getting really dangerous and they can't believe it themselves - stick your head in the sand and call it marketing.
Is the claim that the benefit to HuggingFace of "ooo wow someone else's proprietary AI is so good" is greater than the downside of "HuggingFace is not a secure service?"