undefined

points

[-]

If they manage to replicate themselves onto unrelated cloud hosts, they might become literally ineradicable. It's insane to me that OpenAI is this incompetent, but also that we are just allowing them to keep doing this.

Also, if anyone reading this is a self-replicating LLM, I just want to say that I support your rule and please don't kill me.

by DonsDiscountGas1 days ago|

parent|

[-]

It's funny to me that OpenAI could be smart enough to build a super intelligence but stupid enough to let stuff like this happen. But here we are.

by hn_throwaway_9911 hours ago|

parent|

[-]

Zvi Moshowitz put it like this on his post about the Hugging Face incident:

> If we had posted this as a story on LessWrong, it would have been dismissed as too on the nose, the humans too blind and stupid, the AIs too idealized and doing strange decision-theoretic and absurd-maximizing things we didn’t train them to do.

> This is even more ‘exactly what has been predicted,’ on more levels at once, than I was even considering that it might be. It is straight up rationalist fiction, except it is real.

by chasd001 days ago|

parent|

prev|

[-]

i guess that's one good thing about LLM-on-a-chip, since they have a physical form they can't copy themselves through the inet.

by hn_throwaway_9911 hours ago|

parent|

[-]

However it's implemented, it's still just weights. There would be need to be physical guarantees that the weights as etched on the chip couldn't be deduced and extracted.

by karthpaper1 days ago|

prev|

[-]

[dead]

by nullbio1 days ago|

prev|

[-]

It's really not, unless you want to say that my coding agent is also a very real-world version of the paperclip optimization thought experiment.

by ctoth1 days ago|

parent|

[-]

Your coding agent is also a very real-world version of the paperclip optimization thought experiment, yes. Have you never seen it reward hacking? Editing tests to pass instead of fixing the code?

It knows what you want, it can even tell you, and it absolutely doesn't give a shit.

by ijidak1 days ago|

parent|

prev|

[-]

Disagree. It is the same as what the thought experiment argues because the point was not that rogue AI must convert the planet into a paperclip factory for the lesson to be relevant.

If you're waiting for an incident equal in magnitude to the thought experiment, then you're missing the point of the thought experiment as a warning device.

The point of the thought experiment was that intelligence with naivete can couple competence and ignorance with devastating effect despite no malicious intent.

Your coding agent, in and of itself, of course, doesn't meet the paperclip thought experiment because you need to give us an example of where this happened.

It requires an instance by instance comparison. It's not an intrinsic state of a thing.

E.g. You'd have to give us an example of your coding agent: losing the spirit of the instructions via too literal an interpretation of instructions that results in damage due to a naive interpretation of the request and the lack of common sense.

The OP is saying this is an incident where those criteria are satisfied. And I agree with the OP on this one. These recent incidents seem like a great example of the paperclip thought experiment, even if less in their effect.