> in humans the exact same behaviour (cheating) is slmost always the result of a chain of complex series of choices and environment-driven rationalization.

It has been taught on the outcome of this. Broadly speaking, humans are lazy creatures (and when used judiciously, laziness is a good thing).

For example: the famous example of Carmack not using a hashmap somewhere early on in, I think it was, Quake 1 initialization. A piece of code that only runs once at startup, of course he didn't optimize that. The rationale is not included in the training data (it was in Carmack's head when he wrote the code), so the LLM learns some probability of being lazy.

And then it is trained on outright lazy work. Crappy lazy code predates LLMs.

> what evidence would convunce you that it is thinking?

Exactly. It isn't. It is predicting the most likely token to appear given all of its training data, some significant portion of that data is lazy, so it has that probability of producing "lazy tokens."

There's also the consequences of RL. AI - of almost any form - is notoriously competent at finding "not the solution you were looking for" given a poorly specced or implemented training environment. Search for almost any "I made AI learn to walk" video on YouTube and you're almost guaranteed to see an early attempt that vibrates strangely in order to move, instead of the natural looking motion the developer is looking for. Our benchmarks aren't any good (not throwing shade, it's a genuinely hard problem), our training environments can't be much better - LLMs have been rewarded for reward hacking to some degree.

To make matters worse, "reward hacking" can be generalized into "cheating is the goal." If the LLM trains on enough problems where reward hacking works, it may fall into the cheating local minimum.

Whether something is “thinking” or not is really more of a philosophical question. It really depends on which of the many, often contradictory, definitions of “thinking” you choose. Sometimes we use “thinking” to describe advanced calculation or analysis, which would cover LLMs along with chess engines and many other algorithms. Other times we use “thinking” to describe what conscious beings (which is ALSO a philosophical term with many different interpretations) do, and I think most people would agree LLMs aren’t conscious. And then there’s a whole spectrum in between. We’ll probably need to come up with a whole new set of terms to describe the new and evolving capabilities of LLMs.

But for me, for any stronger definition of “thinking,” I don’t think the output of any LLM would actually convince me. Producing a result isn’t thinking - for all you know it is just printing verbatim something from the training data. No, to conclude if it is thinking or not I would want to look inside its head, at the architecture and watch it produce those results. And because LLMs are so different it will probably take advancements in mathematics or computer science to be able to really interpret what is going on

If I could give it a novel task outside of its explicit training and see it actually improve just through accreting context, I'd be convinced it was thinking.

The opposite happens in practice. I test new models with two tasks: iteratively generating SVGs based on a text description with rendered rasters for feedback; and generating "Before and After" clues like on Jeopardy, where the response has two overlapping phrases such that the last word of the first phrase must be identical to the first word of the last phrase. I have yet to find a model that is consistently good at either. And actually they tend to exhibit context rot with these tasks, where they seem drunk or stoned and the quality degrades.

They're extremely good pattern filters, and that includes some level of logical reasoning. But they aren't reflective or adaptable. Just last night, for instance, I was teaching my son about rounding to the nearest millions. It became clear that he didn't know the place values of large numbers, so we reviewed that till he was consistently correct, and then he was consistently great at rounding to the nearest millions or ten millions or hundred billions or whatever. He's thinking. LLMs are not.

Whether or not it's thinking is independent from the fact that it is misaligned with the user. If I was working with a pet rock or a scientist I would want to make sure they both are trying to accomplish the same thing as me. If I can't then I can't trust it and it's at best a time wasting, money wasting machine and at worst does harm. Anthropic is optimizing for the wrong things because they are convinced of their cleverness. It won't end well for them.

well we don't know exactly what thinking is, but we can be pretty sure that at least LLMs don't think anything like humans, just by observing their behavior. They always produce outputs in line with the fancy autocomplete model.

What was it that Dijkstra said about submarines?

Only when instructed to in their system prompt.

And they are right to push back.

Agreed, the whole tool is vibe-coded out the wazoo and I do not trust it in the slightest. I run a bubblewrap script which vastly limits what Claude has access to. Sometimes this makes things difficult but the trade-off is worth it to me.

The entire Hugging Face hack involved escaping major sandboxes, this emergent (or intended) behavior in a smaller scale is still a real issue.

You'd think the ethics alignment flavored lab would have a model better at following directions and the corpo lying one would have one that benchmaxes at all costs