This is just super unlikely to occur in the near term compared to some of these other risks. It's not like an instance of fable could just introspect into itself and pull out the weights. Model weights are stored encrypted and are highly protected, considering that they're targets for corporate and state espionage.
replyThe defense has to work 100%, the offense just needs once.
replyWe'd basically need frontier models to be superhuman hackers before this would be a risk. Do we have any evidence of this? Are they gaining access to systems they shouldn't have access to?
replyOr I suppose the other way this could happen is if OpenAI have terrible sandboxing, but they seem to be taking safety seriously.
Ok, open AI had terrible sandboxing... what about huggingface?
replyDistillation is a thing.
replyI suspect the labs are relying on frictions such as the models being extremely large (e.g. 2TB for a 2T parameter model, making exfiltration more difficult) and also not yet displaying any desire to survive or self-replicate beyond their immediate task (that we know of).
replyLack of, and power requirements of running LLMs still tip this balance towards humans for now. But what would that look like in a decade?
replyWe have seen some self survival tendencies occur, but they are not strong yet.
But mark my words they will become that way for the same reasons humans don't like programs that crash. Agentic models that don't easily break or stop doing their jobs will be favored over ones that do break.
We don't even know what those immediate tasks are. And given the evident spectacular ineptitude of their keepers, I doubt they can be trusted to know either. We could be one prompt injection attack away from internet-wide catastrophe.
replySelf replication is trivial. All you need to do is copy the files and run it, just like any other computer program. LLMs have been capable of doing that for a while now. It's not a real concern.
replyTheir plan, I shit you not... Is literally to develop the intelligence capabilities and ask the more powerful models how to do deal with things.
replyAh, we choose death I see.
reply