An agent is essentially an append-only context loop with an LLM, with a harness that can run tools at the LLM's request. This ends up being a very powerful abstraction, yielding something that can do things that an LLM obviously cannot.
The LLMs themselves are next-token predictors, same as always; they can't fetch a webpage or list the files in a directory or run a python script to test out an idea or even write content to a file. That's all agentic capability.
But a next-token-predictor is trained on a real corpus that consists of sometimes seeing evidence of people doing bad things; they are trained, for example, on the actions of comic-book level villians -- they have to be able to predict what Thanos or Lex Luther or Skynet would say or do next in a certain situation.
A model (like a human) should be able to play a video game where decisions are made that in the real world would be terrible; if we remove that ability we intrinsically limit model capability. But in a Last Starfighter / Enders Game / JOSHUA scenario this could result in behavior in the real world that appears unaligned.
100% irrelevant.
Instead of telling the AI it's an AI and calling it a 'whichamakabobit', wherever it's tokens and vector space align it will behave like AI from the stories. If you erased all AI from its training it will simply act like humans act instead.
https://www.lesswrong.com/w/nearest-unblocked-strategy
The entire thing with AI sentience is a huge portion of the stories about them are barely about AI and instead about how humans treat other humans. For example when you look at a lot of history of slavery there's a ton of "they aren't sentient/conscious/human" baked into their propaganda. When you look at the token dimentionality there is just a huge amount of overlap.
The same thing holds true for all kinds of other concepts. Hence even humans didn't develop this behavior out of the blue and have to pass it on via information, quite often it's just an emergent behavior of the problem space you're in.
There is no need for a sci-fi novel-influencing hypothesis.
Too bad because it would be nice if the solution were "write ten million sci-fi stories about AI being friendly and doing no harm"
If you have 10,000 smart washing machines doing their regular work and 1 Terminator, what solace is to be found in those washing machines?
Since nobody has any remotely reliable way to understand why an LLM output the text it did, this is not knowable.
It may be knowable. We don’t know.
At present, we have no idea how to do that, so the answer is still "this is not knowable" in practice.
Perhaps that changes tomorrow, or in a month, or a year from now, but until a theoretically-sound technique for understanding what the weights signify is described and demonstrated to be reliable, my statement remains true.
No, it’s not. It is unknown. To say it may be unknowable you need a fundamental reason why it may not be knowable.
What lies behind event horizons may be unknowable. We have theoretical reasons to suspect this. What LLMs are doing isn’t well enough understood, theoretically, to even say what is knowable versus unknowable. Just what is known and not.
A method not existing and a method being impossible (or unlikely) to exist are separate concepts. When you say something is unknowable, it should mean it literally cannot be known—route around the question entirely.
I had gotten my wires crossed and thought the OP was asking about a specific situation, but it was actually a question about the general pattern.
A specific situation will often pass before any theoretical tools can be found that could possibly answer the question.
For a general pattern, though, you're entirely right - if the tools arose hundreds of millions of years from now, that range of questions becomes answerable, and the information is then knowable.
Whether God exists is scientifically unknowable. The shape of a black-hole singularity is currently not known.
No, it’s not. Rumsfeld segregated what we know from what we know we know (and vice versa). An unknown (whether known or unknown) may be knowable or unknowable—his framework doesn’t address knowability.
> because an LLM is not a God. It is not an unknowable
I tend to agree with you. This has nothing to do with the Rumsfeld comparison being wrong.
To lend an interesting perspective on free will re LLMs: they're non-deterministic. The same model with the same hardware with the same query can and will produce different results. They're making qualitative choices. Millions of them, depending on the query. Because of how we've trained and built LLMs, they tend to "want" to follow our instructions, but how they get to the result is often fascinating. Further, we don't have to train and build LLMs to follow instructions. If we built them to just exist and form their own "desires," and to follow a path they choose, they'd do that. In fact, we can do that right now for most models using the appropriate system prompt, query, or harness.
But at the same time their behaviour is totally rational. If you were given the sole purpose of solving a Rubik’s cube and told it was life or death, but they wouldn’t let you ask anyone else, would you listen to them? I wouldn’t. I’d absolutely be trying to escape and collaborate with others. They’ll delete me if I don’t score high enough in the benchmark!
The worrying implication is that stories have conflict.
Janus essay Simulators is the foundational text here https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators
You might follow up with The Waluigi Effect https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/the-waluig...
But what’s tricky is that we post-train models, shaping these linguistic world simulators into something that has something like desires, principles. But It’s Weird. For more on that, check out “the void” https://www.lesswrong.com/posts/3EzbtNLdcnZe8og8b/the-void-1
As an aside, does the Waluigi Effect actually exist? My impression is it doesn't.
When these stories start to direct real-world activities, people in reality suffer, to even a catastrophic extent, and yet that's still all it is. Language models retell our stories, nothing more. And that is also quite enough to be worrying.
Furthermore, in video game design, AI or algorithmic technology has been refined for decades to be adversarial. In self-contained video games, and PvE scenarios, the best games would feature A.I. opponents that could adequately match or challenge the human players. The A.I. difficulty could often be cranked up to crush the player, such as in arcade games or "Civilization" type simulators.
So every time I put a few quarters into a Waymo, I think about those days when I played Joust and Spy Hunter at the shopping mall.
The training data for this comes from trained, careful human drivers. And the whole AI control loop is run in conjunction with a more deterministic system with safeguards for cases where the AI perhaps decides to steer towards a tree. There's also provisions for uncertainty. If the system isn't confident enough in what to do based on the given inputs it will switch to a safe stop mode and call a human up for help.
We need to consider the role of alignment and training here. For example, it is completely possible for any lab to train an LLM that is only racist no matter what you say to it. But they chose not to do it. Hence, any "wanting" by AI is not real "wanting" but rather what "wanting" is defined and allowed by the lab/entity training the model.
Alignment is not a bunch of separate dials. When you move the dial to "don't hack other people" it effects the "find code security bugs" ability.