Fair.

What I am trying to get at is when people talk about "intelligence," beyond academic debates such as this one that mentions qualia, they are talking primarily about value. How intelligent the thing is is how valuable it is (often directly, in an economic sense). My argument is that value is a relational thing that must involves human feeling, intelligence actually has very little to do with it.

When a child slowly reveals to me something he's judged insightful about pokemon, it's valuable to me, even though I'm not learning new facts. When Claude does the same thing, it's not even "not valuable," it's wholly outside of value, even if I do not already know that fact or thing. If ever Claude reveals to me an insight, that insight certainly came from the training data, not from Claude. The soul came from a human being external to Claude -- the training data -- and passed transparently through Claude. Claude is a translator: it feels nothing, adds nothing. It colors the noise. I know this because the facts are the same but the aesthetic structure - how many bullet points, whether it says "delve" or "load-bearing" does change over time, according to the whims of whoever is in charge of post-training.

But the "real intelligence," is the same thing as "real value." It is the soul of the humans in the training data, the books, newspaper articles, etc.

A clever hacker news commenter might argue, well, what if we gave Claude a humanoid body and senses and let it interact with the world, then would it be intelligent? To that I would say, why waste your time and effort? You can get it for free: just talk to a friend, a neighbor, or a family member.

An observation: You can never insult an LLM, but it can certainly insult you.

You can not insult it because it does not care, because it does not have "feelings". But you do.

I agree, and yet it is reasonable to ask if these notions of "context" (sorry!) - namely feelings, external world, body, senses, etc - are somehow distinct from an LLM's notions of context. Today they certainly capture different things, but given the right representations, why couldn't these human notions also be captured as "context"?

The idea of memory does not seem to resolve this: if you allow the machine to "compact" its context, then you've given it a system which is analogous to our own evolving state. (Though this is undoubtedly still less expressive and meaningful than the one we have evolved as humans.)

One idea I've wondered about is our human capacity to induce subsequent mental states: I can effectively decide how I want to feel and take actions to create that feeling. It's not clear whether models exhibit any degree of privileged introspection into their own states. Is this important? I don't know. (Non)determinism also does not seem to resolve it; it's my understanding that there are plenty of philosophers and researchers who think that human behavior is deterministic, or that the question of determinism does not matter.