To be fair, almost everyone who claims LLMs are conscious tends to claim that they are conscious in exactly the way that humans are, to the point of stating that human brains are also just complex next-token prediction machines with a random seed. It's basically religious arguments on both sides.
I have seen people say "you're a next token prediction machine" but only in a similar way one might say "you're a cup of old lard". Not actually meaning it literally.
I have seen people interpret the request to show that they are not next token prediction machines to be a claim that they are, but this is almost always an argument to show certainty is difficult in this area.
People like Hinton have declared that they believe them to be conscious, but clealy indicate that they do not mean just like us.
And it seem obvious to me that language behavior does differ significantly between humans and LLMs based on the frequency and nature of failure states. LLMs routinely hallucinate, or get "AI strokes" or get obsessed about not talking about goblins, etc. This isn't typical language behavior for humans unless they have severe neurological or psychological impairment.
People tend not to "spew words out without thinking" and certainly not all the time by default - we call that glossolalia and (outside of some fringe Christian sects) it's considered a "bug" not a "feature" of the human brain. Human language by default always has intent behind it, even if that intent isn't readily apparent to the speaker. People can recite by rote memory, but that isn't blind token prediction, it's the neurological equivalent of muscle memory. People can have conversations then forget about them because their attention was focused elsewhere, but that doesn't indicate that they were simply "spewing words out without thinking" at the time.
People imagine details all the time. Eyewitness testimony is notoriously untrustworthy.
Our brains seem wired to confidently fill in gaps. We all have a literal blind spot we aren’t aware of because our brains convincingly lie to us and fill in the gap.
I don’t know what an “AI stroke” is, but I’ve definitely seen human beings in good health be in the middle of talking and suddenly forget what they are going to say.
> People tend not to "spew words out without thinking" and certainly not all the time by default - we call that glossolalia and (outside of some fringe Christian sects) it's considered a "bug" not a "feature" of the human brain.
Glossolalia is spouting gibberish, not comprehensible speech.
Kind of weird that you speak so confidently when you don’t apparently know the difference between steam of consciousness and “speaking in tongues”. Almost like you’re AI hallucinating.
This sounds like a description of a child who has not learned to read yet. You ask a child who is not aware of the alphabet and of "words" how many r's are in strawberry you'd get a non-sense answer too. So what you're really pointing out is that the LLMs have not been trained on "the english language" and how words are constructed and what they are composed of. That they operate by tokens that don't correspond to words or letters is irrelevant as an answer to why they can't count the letters in a word. It's not that I know how many r's are in strawberry because of how I'm understanding the word "strawberry", I know how many r's are in strawberry because I know how to spell strawberry. The LLM needs to be trained on this the same way someone who is learning to read would be trained on it. No one should be surprised that an LLM can't "read" in the same way no one should be surprised that a child can't "read".
This interpretation takes things too far away from how LLMs are constituted and so misses important explanatory power. The issue of counting letters in a word isn't about an ability to spell, it's about the nature of one's perception. We perceive words as sequences of individual letters. LLMs do not. I can ask you to tell me how many r's are in some nonsense word sequence and you're fully capable of doing that. LLMs do not see sequences of letters so they are intrinsically at a disadvantage for this kind of question. But this says nothing about its capacity for intelligence anymore than not naturally being able to distinguish frequencies of photons hitting your retina has anything to say about human intelligence.