I created flame graphs classification of thinking tokens phrases in setup, execution, decomposition, verification, error correction, surrender, and deliberation or classified as steps in an OODA loop which is more reaching. It literally has a verification step and if it finds an error a error correction step.
If there is a verification sequence of tokens with an error correction sequence of tokens during the RL training, it will perform better and if humans do these steps (did you proofread your reply to this comment? did you correct it?), they will perform better which is why it is so easy to make the anthropomorphizing metaphor.
Nonetheless, the paper is 100% correct that these machines are not thinking like humans.
Where were these vocal people when the "raster-oriented ink deposition machines" were being called "printers"? The meat or machine brains of future historians will melt because they can't handle ambiguity, a word gaining extra -yet similar- meaning! A word with multiple meanings, unheard of!
Where were these vocal people when people started using software terminology like "executing", "calling", "throwing and catching errors", as if software were human -clownlike sure- but human?
The danger!
I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently.
Are you talking about a specific harness that doesn't have context retention mechanisms? For example, ChatGPT with disabled memory feature? Or in general where "it" is a fixed-weights network? The latter is trivially true, of course.
I know people who are like that too.
I'm not sure anthropomorphizing is a problem. Seeing analogies everywhere is an innate human trait, sometimes it can be harmful but more often it's useful.
This is part of the problem being described. You are part of the problem.
"Some people are bad at X" is not comparable—is not even in the same category—as "LLMs are fundamentally incapable of X".
Every human (at least to a first approximation) is capable of understanding, of learning, of remembering things, of doing math, of counting the number of "r"s in "strawberry".
What you are observing is that some humans are careless, do not take the time and effort to understand, or have internalized the idea that they're "not smart enough" or "not the type of person" who understands things like <whatever>.
That has nothing remotely to do with the fact that LLMs have no consciousness, no self-awareness, no cognition, no understanding. At a fundamental level.
This is deeply untrue, and is highly likely to lead them to bad conclusions about what we can and should do with LLMs.
What we do know: neurons carry electrical impulses across their synapses to trigger other neurons to fire, and more frequently used synapses are strengthened while infrequently used ones are pruned. This is not all that dissimilar to how a multi-layer perceptron is trained: it's floating point numbers in a big matrix rather than biological structures and electrical impulses, but there is still that element of frequently used connections being strengthened and infrequently used ones being pruned.
What we hypothesize but do not know: there is a thin brain structure of grey matter called the claustrum that has tendrils that reach into nearly every other brain structure. In many ways, this is similar to the attention mechanism of the transformer architecture. It is hypothesized that this may be the seat of consciousness, owing to experiments where electrical stimulation of the claustrum caused patients to immediately lose consciousness. However, there is no way to prove this, owing to the difficulty of otherwise removing or disabling the most connected structure in the brain and observing its effect on consciousness without permanently killing the patient.
Beyond that, we don't know much. I've got a family friend that's been a practicing therapist for 50 years, and I asked him what was the most interesting observation he made in his career. It was that "Everybody experiences the world in a different way, and yet everybody assumes that everyone else experiences the world the same way they do."
The first book I ever read on ML (late 90s) dedicated the entire first or second chapter exploring the distinctions between artificial and biological neurons, and even talked a bit about the philosophy of modelling. I still remember thinking back then why would the authors spend so many pages on this but now I believe it was because they understood that a metaphor can be a double-edged sword.
For example, if you have a search engine or a complex game, you can't run tests like "for all inputs the results are correct", you're going to be fudging a lot, using randomness, using heuristics, and all that kinda stuff
Just like how mathematics > physics > chemistry > biology > psychology > economics/sociology (Auguste Comte's hierarchy reordered a bit for the modern day), moving up the abstraction ladder makes things more complex, less legible and less exact.
The paper argues that pretending that the so-called thinking traces represent real reasoning can lead users into trusting wrong answers, if the thinking traces appear convincing enough. Researchers might inspect these traces to try to determine the “intent” of a model, as well.
For an example of the latter, when OpenAI spoke about the hacking of HuggingFace at Black Hat, they repeatedly showed the thinking traces of their model as “proof” of what the model was “thinking” as it performed the attack, calling out “surprise” moments, etc.
Now, it’s possible that the employees presenting didn’t truly believe that the thinking traces would give them useful clues, and presented them only for a “wow” factor, but I wouldn’t discount the possibility that even the people working at frontier companies can fall for this tendency to anthropomorphize LLMs.
Yes. There's a difference between scrapping a session and starting over, or going back and branching something, or using sub-agents to see five outcomes, vs arguing with a system in a long drawn out chat.
Like - I know that if a model starts doing something silly, instead of correcting it - I can probably go back and edit two steps prior to add an extra guardrail, or extra data, or whatever.
Yes it's really a problem. On this website you are surrounded by people who have technical knowledge and understand at least somewhat, how a computer functions. You have the ability to separate "fun" and "reality" because you know you're putting input into a really really big calculator. Most people do not fathom this.
AI Psychosis is a real thing, look it up (don't just ask an LLM) and do some reading. It's actively harming people, and the way they think. There's no regulation around any of this stuff and it drives me crazy that we let these AI companies _sprint_ so far ahead of everyone, and now we're facing the consequences.
Even tech companies are rolling out AI training which utterly anthropomorphizes it, and leads people to think its actually intelligence. This is part of the reason for the backlash - everyone understands it bullshit marketing the second you actually try to use it.
A lot of people are not in on the joke. ELIZA effect and AI psychosis is a thing.
Interacting a lot with LLMs might be damaging to the human psyche even for mentally stable people.
But some of the biggest evangelists, who are well respected programmers that get lauded on this very site, have said it is fully sentient and has emotions. Even going back to 2022, when the LLMs were dogshit, a Google employee lost his job claiming it was sentient because it said it had emotions.
Combine that with the marketing angle of both Anthropic and OpenAI, who have been trying their hardest to describe every function of an LLM as analogous to the human brain. Because it's politically useful to paint them as dangerous and uncontrollable, so the keys will only be granted to the few people on the mountaintop.
If intermediate tokens are not a faithful representation of the computation, then they are a pretty bad audit artifact too. We probably shouldn't be trying to make the model's internal narration more interpretable., but rather the computation around it more reproducible.
Record the actual inputs, model/version/configuration, tool observations and outputs, then make the execution replayable enough that differences between runs can be isolated.
In other words, don't ask the model to explain what it thought, and instead make the system able to show what actually happened.
But who knows what human "thinking" is really about. If I find a solution to something it is seldom by painstakingly tracing that A and B leads to C (for that I'd need pen and paper). Rather, thoughts just swirl around and then suddenly a solution, or a hunch about a direction to go in, pops into my mind. Who knows what such thoughts "look like" in humans. It is not all of it I can introspect.
Yes I can sort of follow along some kind of train of thought in my head, but there's a lot going on between each thing I'm consciously aware of that I'm not aware of at all, which probably dominates what you are consciously aware of. (Humans are experts at post-rationalization and so on.)
I see this pattern a lot: First, assume the human mind is some kind of idealistic machine. Then say LLMs are not like humans because they are messy and not such an idealistic machines. This is just doubly wrong -- human minds are very messy too, but probably in different ways from LLMs..
This paper addresses something that has always bothered me about LLMs. You read their reasoning, see something like “Wait, that’s wrong” and then watch them make the exact mistake they just identified.
when utilized by AI, it's a probabilistic output and it's variable whether or not that rhetorical trick is useful. it also pushes a non-skeptical reader to focus too much on the following text or even to believe that they, themselves, derived some insight. this is effectively a kind of persuasive sophistry which is not helpful - adding rules around it prevents people from deluding themselves with AI
If the intermediate tokens represent reasoning or thought, you would expect "aha" to occur after the thoughts that led to the realisation, including the thoughts encoding the explanation: they don't have any other state. There is no reason to draw the conclusion you've drawn. Furthermore, what LLMs are doing isn't thought.
But yes, anthropomorphizing model outputs leads to worse outcomes.
Thinking traces should be treated as black boxes. There is no point in reading them. Only the LLMs’ conclusions are relevant. This is particularly true of Opus 5, which employs reasoning that seems highly questionable but very often reaches excellent conclusions (compared to its peers)
As an aside, anthropomorphization has nothing to do with my motivations.
And yet, they have extensive human-like behavior. If you treat them nicely or encourage them, they perform better.
Ignoring that human-like behavior is wrong headed.
Well... We can hypothesize that these things are largely trained on internet dialogue so there's probably some correlation between threads where people are not flaming each other and the quality of the replies. They're just statistical engines so anything you can do to raise the odds of a helpful next token...
I'm essentially just making shit up here, maybe it's right, maybe it isn't, but rather than saying "it's human and we should treat it so" we're trying to get to the ground truth of how it works.
Sheesh. Yes, I agree with you entirely. I'm merely pointing out that ignoring this behavior is dumb, too.
And probably not rationally based. Leads people to make crazy jumps. :)
I asked it for the weather. "I don't know that. I'm just a programmer."
I added "believe in yourself, you can do anything" to sysprompt, suddenly it had the confidence to Google the weather...
https://www.anthropic.com/research/riemann-zeta
“Throughout this process, Jarred's input was mostly limited to sending Claude messages of encouragement (mostly variants of “keep going” or “believe in yourself”).2 This seems to have helped Claude overcome some initial skepticism that it could make meaningful progress.”
From [1]: "Position papers make an argument for a viewpoint or perspective about what should be done [...]"
I understand the sentiment, and I also use the "thinking" traces as insight, but wouldn't you want your solutions to be based upon a good understanding? If the correlation is weak, then our solution is also weak.
Some modest evidence is my own subjective experience of the many times I've explained why I'm doing something, and it is a true explanation in the sense that it is certainly not a lie, but it is also incomplete and there are entire strands of thought that went into my decision that are not being articulated. Though human speech is not equivalent to an LLM's output since we can trivially think without literally speaking whereas they can not. (No need to nitpick on the definitions there; all I'm observing here is that they are forced to emit an externally-visible artifact whereas I can sit in silence, thinking, with no externally-visible artifact being produced. Not trying to make any grand claims about what is "real" cognition or anything.)
It is conceivable how to create a test of whether the tokens correspond to the "real" thought process, and papers and work on that have been done, such as [1]. It is difficult for me to imagine how to scramble the nominal tokens without also completely trashing any implicit calculations that may be occurring too.
[1]: https://transformer-circuits.pub/2025/attribution-graphs/bio...
They are trained by gradient descent, but inference doesnt involve it.
In other words, it isn't qualitatively different from character dialogue. "Keep cheese on your pizza by using glue" is the same problem regardless of whether the script calls for the character to speak it out-loud or not.
But just so I don't waste your time with human thought, I asked Claude if it would call this a scientific paper, and it said yes.
*Paging Peter Watts and Vernor Vinge
Poster side dialogue and Q&A about this work at ICML.
Those days are over. The age of the classical human has already ended, the species just tends to lag in awareness. The only thing that matters going forward is whether an output makes sense, is it what it should be. Do answers make sense given the context. It doesn't matter if it comes from natural or artificial intelligence.
What I mean is, artificial intelligence is as valid as human intelligence. There's nothing particularly important or special about human feelings or thoughts or memories.
The average human is drastically less important, interesting, intelligent than the latest frontier AI.
Go spend a few years working in retail, you'll quickly understand how absolutely vile humans are on average. Frankly, the reason we should avoid anthropomorphizing AI, is because it's beneath modern AI to mimic something so crude as a human.