> Recent debates have been clouded by a misleading inference pattern, which we term the “Redescription Fallacy.” This fallacy arises when critics argue that a system cannot model a particular cognitive capacity, simply because its operations can be explained in less abstract and more deflationary terms. In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions. Such arguments are only valid if accompanied by evidence demonstrating that a system, defined in these terms, is inherently incapable of implementing . To illustrate, consider the flawed logic in asserting that a piano could not possibly produce harmony because it can be described as a collection of hammers striking strings, or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings. The critical question is not whether the operations of an LLM can be simplistically described in non-mental terms, but whether these operations, when appropriately organized, can implement the same processes or algorithms as the mind, when described at an appropriate level of computational abstraction.
This sounds like a dismissal of the argument through a characterized straw man.
That is, it seems that reducing the complexity of the brain to "collection of neural firings" is not being honest about everything involved to a much greater degree than saying neural networks are a "collection of statistical calculations".
I too believe LLM's will grow in complexity, but presently I can not even fathom how they can be compared to the complexity of a system such as the human brain.
Like driving a car — it's transportation, and it will get you where you're going, but it doesn't use bones or muscles. It has many characteristics in common with builogical locomotion, such as energy requirements, intertia, and the need to navigate, but it doesn't involve proteins or sugars really.
Totally understandable; I don't think we can fully understand the human brain, using the human brain. We can understand its principles (firings and chemistry, structure and specialized areas, etc) but otherwise it's a capacity problem.
And while I can't fully understand myself, let alone another person, I definitely enjoy talking with people and sharing thoughts that I realize I wouldn't have had on my own.
Nobody actually makes this argument though.
https://www.goodreads.com/en/book/show/217432753-the-ai-con
which describes LLMs as "souped-up autocomplete", complex statistics that cannot truly understand anything. A more recent example is this paper:
https://zenodo.org/records/20071869
which says,
> [LLMs], as turbo-charged statistical models (recall their formal relation to logistic regression) can only but provide correlations.
And, of course, the Stochastic Parrot paper is the classic example in this area. It is from 5 years ago, but "LLMs only do statistics / can't understand" is very much alive and active among academics, even if it is a minority position.
cognitive: as in reasonable; of, relating to, or involving conscious mental activities (such as thinking, *understanding*, learning, and remembering)
That term is used to describe mental aptitude or skills, like the ability to learn new languages or do math.
By the way, I know it's a parody of another story that makes this exact refutation. But I think this only serves to highlight the point.
How do you connect that description to "LLMs could not possibly be good models of some cognitive capacity"?
I don't think it makes any sense to say that consciousness is a cognitive capacity. Cognition is one of many qualia that compose the experience of consciousness from the inside, but it's not the only one, and I can easily imagine consciousness without cognition at all.
So I don't think it's weird at all to say that LLMs can be good models of some cognitive capacities (particularly the ones embodied in language) but lacks others, and overall lacks consciousness.
Look, this isn't necessarily directed at you, but I've been a researcher into the theory of deep learning for many years now. I've seen all the phases, heard all the criticism, had to constantly argue against this. Gary Marcus was one of the loudest voices of this argument, but every would-be philosopher came out of the woodwork to explain why LLMs are no more than stochastic parrots because of their design. Geoffrey Hinton famously had to debunk these arguments multiple times.
And now that LLMs start to clearly exhibit intelligent behavior and can be somewhat reliable, now "nobody ever thought that LLMs could not possibly be good models of some cognitive capacity because of next-token predictions, or linear algebra, etc."? No, that's not okay.
It reminds me, oddly, of the debate over whether video games can be "art". A turning point was when they actually did something that art does: [evoke profound emotion and thoughtfulness](https://en.wikipedia.org/wiki/Shadow_of_the_Colossus#Legacy) for the player.
(And before that, "[Can photography be art](https://daily.jstor.org/when-photography-was-not-art/)?")
We may not come to something as simple as "machines can be conscious", but we will certainly have to understand consciousness better if we want to refine our questions.
---
Edit: My point is that we don't need to be angry, but we may have to tolerate people expressing their exploration through overly-confident language, and be patient with that.
And Ted here is obviously exploring. His examination of Claude's constitution clearly shows some nuance. He asks:
> So, given that Claude is not conscious, what are we to make of Claude’s constitution?
And his conclusions are split, between this is useful and this is dishonest. It's a great tension IMO.
> The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter. This might seem like a reasonable goal to work toward; I think we’d all prefer it if chatbots never emitted sentences such as “You should kill yourself.” However, for all the times that “honesty” is mentioned in Claude’s constitution, I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.
So, I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing. There is probably tons of data pattern that LLM can learn from to be able to reproduce a sentence continuation that is convincing without having to learn the specific mechanism that is "conscious".
For me, one element that shows it is the case is the absence of world model (or "human-like" world model) despite the fact that the sentence continuation is convincing. If indeed the only way to produce sentence continuation convincingly would be by "simulating a brain", then it would not explain the first LLM from several years ago (before the extra layers of RLHF, ...). They were able to have quite convincing conversation on a lot of non-trivial aspect, and yet failed on some aspects that should have been basic for a system that would have been trained to work like a human brain. It shows that it is possible to "cleverly disguise examples of sentence continuation" without having to build elements that one expect on a conscious being.
Similar to: "Birds fly, my spinning helical device flies, therefore we've started to replicate how birds fly."
> without having to build elements that one expect on a conscious being
One of the elements I expect in a conscious being is that you can't rewrite it by changing the introductory paragraph.
When it comes to LLMs, almost every "mind" we humans perceive is a fictional character in an LLM-generated story-document, one we are either reading or which is being "acted" at us by regular code. Our own instinct for pareidolia and simulating/inferring other minds is very strong, which means we should require really good evidence/logic to counter our instincts.
Even if one believes the LLM has a single "real mind" as an author of every document... what evidence do we have that it is conscious or "self-inserting" itself as one of the characters in the document?
If we had enough knowledge of the workings of the human brain, you could alter the perception of every single memory you've ever had. And limited versions of this already happen all the time. Human memory is notoriously unreliable for a reason.
Are you aware of the Recovered Memory Therapy Scandals of the 80s/90s ? Boy did that ruin a lot of lives. You can rewrite a human by changing their 'introductory paragraph'. It's just not as accessible.
Knowing how something works is not the same as having the tools to change it.
Discovering memories are incorrect does not massively change who we are. As someone with a very defective memory, I discover on an hourly basis that I'm won't about something I thought was true, but there's still continuity and consistency to my personality and general approach to life.
...in fact, as someone who was raised an evangelical Christian and believed wholeheartedly without a shadow of doubt, then lost my faith entirely in my late thirties, I sort of did have my "introductory paragraph" changed, yet my wife, children, and friends would all say I'm still me, and that my core personality and nature remains largely the same.
> Are you aware of the Recovered Memory Therapy Scandals of the 80s/90s ? Boy did that ruin a lot of lives. You can rewrite a human by changing their 'introductory paragraph'. It's just not as accessible.
The recovered memory scandals are not even close to evidence that you can rewrite a human.
The people who thought they had learned new facts about themselves did not suddenly lose their context as humans in 20th century America.
They did not suddenly lose their sense of humor, or develop a previously-unseen penchant for murdering small children.
They experienced a revision of belief, and a pretty major one that really distressed them, but it did not change everything about them.
LLMs _do_ manifest wildly differently based on the first paragraph.
Understanding is not consciousness.
Their training is all about understanding. There is nothing in their architecture or training that credibly optimizes for rich self-awareness.
Given non-persistent experience, non-continuous operation, no ability to build up generalizations and aggregate experience of their own self-awareness over time, they seem to be structurally designed to not have consciousness.
This is a case where acting is very credible. Understanding of other's consciousness, in a functional and third party sense, isn't a substrate for personal experience.
In stark contrast, humans develop consciousness gradually over continuous time with persistent aggregation of experience. By the time we can recognize our own consciousness in the abstract, and reason about it, we have had it for some time.
My point is that the fact that AI can reproduce convincingly human sentence continuation does not imply that the AI has no choice but ending up using a mechanism that "understand" rather than just have learned data patterns that are very effective to fake human sentence continuation but are meaningless in term of understanding the concepts.
And I think that if indeed the only way for AI to reproduce convincingly human sentence continuation would be to end up in a configuration that uses the "understand" mechanism to do so, the behaviour of the first LLM would not show that they are so good at sounding human and yet so bad at failing basic "understanding" tests.
Taken as an absolute without any addition context you are right.
But we are not talking about abstractions but specific successful models. The number of parameters models they have may seem large, but they are very small relative to the training data that they have to summarize. That cannot do it without discovering that patterns that make sense out of it.
And we can verify that. Simply discuss completely disparate topics, with some kind of intersection. Converge several highly unlikely topics, there are so many it would take billions of years to exhaust unlikely combinations.
If the model is only interpolating it will produce gibberish.
But that isn't what happens.
The fact that models can be near expert, and sometimes expert, across vast areas of human knowledge is a clue. If they don't understand that, then the question is, why do we think people understand things. Does having an answer mean a human understands something, or is their intuition and stream of conscious reasoning also not understanding? To be even handed about what we mean by understanding.
I don't think it's true at all, and I think we have indication that proves it is false.
We have "basic" LLM, the ones from 2023. They were producing _very convincing_ human text, and yet, they were too often failing basic tests that require understanding.
Now, we have more advanced models, but the counter-example of "basic" LLM demonstrates your assertion is incorrect: these model _did_ produce very convincing human text and yet did not make sense out of it.
But for the more advanced models, the problem is that they are "on top" of basic LLM. So, the first step is a training that build a mechanism that produce convincing text without understanding, and then, the "residuals" are fine-tuned. The result is very unlikely to add "understanding" to the model, because to do so, the whole system needs to deconstruct the basic LLM, to go back towards less efficient situations in order to rebuild almost from scratch. The fact that modern LLM are based on basic LLM means that the first step put the cursor in the bottom of the "basic LLM mechanism" valley, which is a local minimum. And any layer on top of it cannot "climb up" the slope of the valley, pass the ridge and fall into the next valley, even if this next valley has a lower minimum.
> The number of parameters models they have may seem large, but they are very small relative to the training data that they have to summarize.
That is demonstrably an incorrect logic jump. For example, CNN are able to distinguish between pictures of cats and pictures of dogs. The weights in these models are very small relative to the number of pixels they have been trained on. Yet, they distinguish cats and dogs by finding specific shapes in the pictures, without understanding what a 3-D cat and a 3-D dog is.
They have done that without discovering the typical human pattern that make sense of "cat" and "dog". And yet, the number of weights is very very small with respect to the number of pixel used in training.
> And we can verify that. Simply discuss completely disparate topics, ... > If the model is only interpolating it will produce gibberish.
What you are saying is that the model is not simplistic interpolation. But that is a straw man argument: people who say that LLM don't understand don't say LLM are equivalent to simple interpolation machine.
But the problem is that you can have very good predictions in novel situations without understanding.
For example, if you have 10 totally different situations that can be described with a Gaussian curve, and that I show you points for a new situations that cover the left side of a Gaussian curve. Then you will be able to guess that the right side of the curve, which is not an interpolation as it corresponds to situations you never saw, will behave like the rest of the Gaussian curve. And yet, in these 11 situations, I did not even say which real physical phenomenon I'm talking about. You haven't understood anything about these phenomenon, all you have done is guessed that a typical pattern that you have observed somewhere else is more likely to apply here too, without even having to understand anything about the reality of this situation.
And of course, this prediction is "a guess": maybe, for once, in this 11th situation, the curve will start as a Gaussian curve but will suddenly be different. But it happens that the reality is that in this 11th situation, the correct description is a Gaussian curve (because, due to the maths, Gaussian curves are really common). So, when you make your prediction, it looks like you understand the situation, it looks like you understood the physical mechanism that applied here. But it is not the case.
So, no, correctly doing such prediction does not demonstrate understanding.
> The fact that models can be near expert, and sometimes expert, across vast areas of human knowledge is a clue.
That is not at all sufficient. A Chinese room experiment will do that despite the system not understanding Chinese. A pocket calculator will be able to be expert in math computation.
> If they don't understand that, then the question is, why do we think people understand things.
That's the wrong question. The correct question is: we know people understand things, and we see AI behaving similarly to people in some aspect, but is this behaviour _requires_ understanding, or can we reproduce this behaviour without needing to understand?
The fact that "basic" LLM were able to reproduce very convincing text that look like they understood X and yet were demonstrably showing lack of understanding of X demonstrates that we cannot just jump to the conclusion that just because it looks the same, the only possibility is that the core mechanism is identical.
Sometimes, a problem being hard means you only get bad solutions, or increasingly accurate ones.
The planet isn't big enough for the proverbial interpolative stochastic parrot, over the training set of global human communication.
Firstly, how do you know that the optimal way to highly compress complex information is to understand it? You think it is obvious because you are very familiar with "understanding" as a way to summarise complex information. But there can be billions of different ways, outside of human imagination, that is as good or even better.
But secondly, LLM don't find the optimal way, they find the local minimum. Everyone who worked with NN knows that they are prone to come up with spurious pattern, incorrect correlations and bad workaround to guess the correct answer. You regularly need to nudge the NN by creating specifically engineered features to avoid them to fall into the first local minimum.
When it comes to LLM, it is extremely complicated to control to see if the LLM has triggered on a misleading pattern that, by chance, links two "tokens" together, or on a real concept that indeed links two "tokens" together. Basic probability implies that there are probably tons of "fake patterns" engraved into the weight during the LLM training, "fake patterns" that should not exist if there was any kind of "understanding" of the abstract mechanism that links these tokens.
You are basically arguing for a functional account of consciousness, but things like this have been debated for literally decades/centuries in philosophy.
The problem with the hallucination argument is (1) that is much less of a problem with good current generation AIs, and (2) living conscious breathing human beings also have a disturbing tendency to make shit up, too. So a tendency to make stuff up doesn't really serve as a disqualifier for consciousness.
Also worth mentioning that the guiding rule of what's philosophical or not is whether it's actually useful. Actually useful philosophy usually becomes something else. Usual some scientific discipline or another. And as it turns out, theories of mind are likely to become extremely useful in the near future. Expect huge advances!
1) Good current generation AIs are specifically trained to reduce hallucinations. If we had new AI system that happened to not have hallucinations as a side effect of their training, then it would be convincing. But here, it looks like we have built a pocket calculator that answer 7+13 = 14, and on top of it, we added a layer that says "if the input is 7+13, then replace the output by 20". This pocket calculator still does not know how to calculate, we just added a layer to hide its mistakes.
2) Not only "make shit up" is not the same as "hallucination" (either "making shit it" is done when the individual knows it is unreliable, or when the individual was given wrong inputs), but the point is not to say "hallucination implies no consciousness", but "large quantities of hallucinations in situations where a conscious system would be unlikely to hallucinate implies no consciousness"
I am far from convinced that the training and inference regimes of LLMs would qualify as “experience” by any sense of the word.
Now, if we hooked up a plethora of audiovisual and tactile sensors with live feedback directly to a neural network rich with transformers, that was always powered on and fully autonomous, we may be getting there. But we’d probably also be on the verge of manmade horrors beyond our comprehension.
Biological rodent neural networks in a Petri dish stimulated by electrical impulses - more or less conscious than LLMs?
Human on life support, unable to respond to any external stimuli, “braindead” - more or less conscious than LLMs?
And, yes, concerns about whether biological rodent neural networks are or are not conscious come up frequently in the biological neural network papers. I'm not sure I would want to be a researcher trying to get an experiment past an ethics committee if my biological neural network had 25B rat neurons. (I would hope that they could not).
There is no independent "consciousness mechanism" that one might imagine humans have learned or evolved for its own sake. Evolution learns various solutions to optimization problems, and so if consciousness evolved then it was either useful instrumentally, or it is a byproduct of some organization that is useful instrumentally. The point is that as a solution to certain kinds of optimization problems, consciousness can conceivably be the solution to the optimization problem of predicting the next token of text written by humans who themselves have complex phenomenology. There is nothing that a priori constrains token prediction from the domain of consciousness.
>For me, one element that shows it is the case is the absence of world model (or "human-like" world model) despite the fact that the sentence continuation is convincing
World models don't have to be rich and detailed to count as a world model. Lower life forms might be conscious but they only model the part of the world useful for their existence in their ecological niche.
Yes, I agree with that. Consciousness is a good way of generating convincing human text.
What I don't agree with is that consciousness is the only way to generate convincing human text and that because we have convincing human text, it can only imply we have consciousness.
There is a huge probability that generating convincing human text can be done without consciousness. Either because there are efficient mechanisms as efficient as the way the human brain deal with this problem and that the LLM found one of them (and these mechanism may be quite difficult to imagine for a human). Or even because the LLM found a local minimum and is stuck there.
To re-use the evolution approach: evolution solved the "flying problem" with bird feathers, but also with insect wings or bat wings. The fact that evolution ended up using feather does not imply that everything that flies can only fly with feathers.
> World models don't have to be rich and detailed to count as a world model
I agree in general, but here, we are talking about machine that reproduce all human language. The argument I'm answering to is pretending that "all of human knowledge" is understood, which include every single human concept. This has to be everything, because LLM is able to provide convincing text about every subject. If on some subject, the LLM is able to provide convincing text without "understanding" it, then the argument that it is impossible to provide convincing text without understanding it collapse.
> There is nothing that a priori constrains token prediction from the domain of consciousness.
We don’t know either of these are true or false though. We simply don’t know. There is no agreed upon definition of consciousness, aside from maybe _the having of qualia_, so arguing that some can or cannot be conscious a priori can’t be done.
No one genuinely engaged with the topic is confused about the target of the term (phenomenal) consciousness. Definitions come once the theoretical work is complete, to be articulated as part of a fully worked out theory. The lack of a definition doesn't prevent us from investigating the subject or offering conjectures. What we can do is offer a precise description of the target and argue for or against whether LLMs reach the description. We will of course debate whether the offered description captures the relevant phenomena. But this is all just part of the process.
Try to define qualia though without explicitly or implicitly recursing into consciousness.
It's all a large house of cards that's built on handwaving and "I know it when I see it".
sometimes humans making claims about AI intelligence or consciousness also identify spurious patterns that do not correspond to the problems of intelligence or hard consciousness.
CNN are finding patterns, sometimes relevant, sometimes spurious, but I don't think people argue that CNN have evolved consciousness or understanding of what a cat or a dog is.
Here, the argument is "LLM are able to understand, because 'understanding' is the only pattern to reach the goal". I'm saying that it is unlikely to be the only pattern, and that it is likely that they find a local minimum on a system that reaches the goal that does not use 'understanding'.
The reason I'm saying it is likely is because "basic" LLM shows behaviours where they are producing convincing human text and yet doing things that are really difficult to reconciliate with the fact that they have understanding.
(And before that old argument is used, yes, I know sometimes some humans fail to understand. The problem is that the majority of humans don't fail to understand basic stuff in the majority of the time, while the "basic" LLMs do. The fact that you roll 10 dices 100 times and 1 of them never land on 1 does not convince me that that set of dice is loaded. The fact that you roll 10 dices 100 times and 9 of them never land on 1 does convince me that that set of dice is loaded.)
That reminds me of a niche paper [0] critiquing a certain way of teaching remedial math that was over-focused on tests. A kid named Benny (12) was building up (wrong) "rules" for math which still somehow gave enough of an illusion of progress in terms of test scores that his misunderstandings hadn't been caught earlier.
> Benny was able to explain his procedure; e.g. for 5/10=1.5, he said: "The one stands for 10; the decimal; then there’s 5... shows how many ones." In another example, 400/400 = 8.00 because "The numbers are the same [number of digits]... say like 4000 over 5000. All you do is add them up; put the answer down; then put your decimal in the right place... in front of the [last] three numbers."
If the tokens didn't correlate to words imbued with meaning outside the system, if the LLMs were trained on patterned data that had no meaning to humans or something there wouldn't be any conversation about these things being conscious at all.
Turing complete systems can be built out of matrix multiplications, out of attention, out of key/value lookups. The Chinese room is Turing complete. By claiming it cannot understand things because it is built out of components computing devices can be built out of, we are claiming no computer can because no computer can. This is a very bold claim indeed, and also we’re assuming the conclusion! The claim is no more convincing than “brains cannot understand things because they are made out of neurons”. The system may or may not have some particular properties, but we have to do more work than just gesturing at the components the system is made of when making claims about it; the alternative is, at best, a world where we prove too much and conclude that humans, too, are not conscious.
For starters, we need to pin down the terms under discussion enough that they don’t just mean whatever we need them to in the moment.
Exactly.
As Ilya Sutskever has pointed out, if you read a mystery novel up until the reveal of the culprit, and then fill in "The killer was _____", don't you need to understand the novel to accurately predict the next word?
Language is tremendously complicated. "Time flies like an arrow, but fruit flies like a banana." "Hard hats must be worn on site; dogs must be carried on escalators", etc. Predicting the next token requires understanding, full stop.
> if the rules are followed, no understanding is neccessary.
The rules are the understanding.
(Note that understanding != consciousness)
The article also makes this assertion that it replays everything over and over again to create each character one at a time as some way to demonstrate the autoregressive self attention mechanism but it’s really not accurate at all, and it trivializes what is going on.
I’m am not asserting LLMs are aware or conscious that’s on the surface profoundly absurd. And I do understand your point that the fact it emits in words something that seems to speak to us gives to the air of humanity that’s isnt real. However there is a very real emergent reality that our language alone appears to lead to embedding a form of thought and understanding that is latent in our use of language in communicating that is in fact coming through the model. It is not regurgitating its corpus and pattern matching because the patterns you input and it emits are not where the inference is operating, its within this enormous vector space through these complex non linear activation functions with learned residuals not in the language corpus.
It is not conscious or aware. It is something else, not human. But if you can not see it as amazing you have lost the capacity to dream.
You, of course, wouldn't notice if your only experience of LLMs was chatting with the cheapest, smallest, least capable LLMs that you get through ChatGPT, or Google search.
It becomes pretty obvious when you use a coding AI on a daily basis. It is the context buffer in which the magic occurs, not the tokens that get spit out one at a time.
Every day, I watch my coding AI develop plans, search the web a half dozen times for documentation, grep through my entire codebase looking for pieces of related code and context, analyze relevant source code across multiple files, spit out an initial plan for implementing the fix before starting to execute it, run requests through some sort of advanced mathematics tool (they are EXTREMELY good at graduate-level calculus and linear algebra), implement fixes that extend across half a dozen files in 2 different computer languages (typescript and C++), run trial compiles and fix coding errors in its output, sometimes developing sub-plans to deal with compile errors. I've seen it get halfway through a fix and revise its initial plan mid-flight as it encounters something in existing source code.
Not vibe coding, to be clear. Targeted use of a coding tool by a by a professional senior software developer with decades of experience, and fair bit of expertise with the limits of what sort of problems my coding AI can and cannot do. Every line code reviewed. Sometimes it needs additional prompts, telling it how it mis-implemented something, or specifying more carefully what I actually want but didn't properly express in the initial request
All the time maintaining that context across multiple request, so that I don't have to restate requests from scratch.
A particularly interesting revision: "You have misread the equation (13) on page 112 of 'Spice, the Manual 2nd ed.'. I should be ....". (It had previously identified the textbook as a source I was using, from comments in source, in a preceding request, and actually already read cited pages in the PDF file, which it had found online). And I had actually asked it to implement equation (13), which was, in fact, badly typeset. The error it had made was defensible, if not the best reading of the equation.
"You are correct. Let me fix that." (producing updates to the implementation of the equation in code, AND code that implements the symbolically-differentiated version of that equation 60 lines later, which is not explicitly given in the text). The text says "take the lagrangian of equations (11), (12) and (13)" or something like that.
ALL information that gets carried in context buffers, even though it's generating code one word at a time. The bulk of the magic occurs in context buffers, not spitting out words one at a time, which, for my coding AI is, I think about 250,000 tokens.
I think it's pretty safe to think that my coding AI is working out of context buffers that may carry plans and research results consisting of tens or hundreds of thousands of arranged tokens carried in context buffers through the multiple steps of the implementation, and later revision. None of that would be possible if were simply working one token ahead.
I kind of suspect that a lot of activity occurs in the first few words of its response. "Let me examine your current source code and develop a plan. Ok. I can see on line 131 where you want me to implement the equation.". (An opportunity to perform about 27 updates of the context buffer). And in the sometimes hundreds of lines of output it generates as it talks itself through what it needs to do.
Sure, it's the best we have online, but that does not make "the internet" the sum of all human experience. To reduce all of humanity down to the text on the internet is reducing us to the level of machines to fit the requirement of what a machine can process / simulate.
I think the main complaint is LLMs don’t arrive at the answer the way we do. It’s capable of emulating some of our behavior but not all as the mechanism by which it works is very different.
Maybe I’m wrong about this but one thing humans do that LLMs don’t is deductive reasoning. LLMs seem to operate entirely of inductive reasoning.
This isn't an argument against their understanding things.
But I expect you are right, that their understanding may have major different qualities from ours.
Along with significant commonalities. (They don't reason via stream of consciousness in a way alien to us.)
Except this is not consciousness.
It's a great interview, if you're interested: https://www.youtube.com/watch?v=NgDIG8u1-CA
That said, I'm not sure I follow what you're actually asking here? I'll also note that I'm not taking a position one way or the other, just sharing a podcast and noting that an extremely reputable scholar on the subject of consciousness seems to have a bit more uncertainty and humility than many commenting here. ;)
I'll find time to listen to your link, it sounds interesting. My objection is the strange idea that humans are automatons that are keyed off input like a clockwork machine and operate sequentially. This is clearly not the case.
I'm not sure that's a compelling argument. Humans can be put into a similar state where they are unconscious and not thinking. Think of someone in a coma, for example, where we actually measure and confirm that there is no brain activity where they're in that state.
They are not actively conscious, but that doesn't nullify their consciousness from when they were awake, right?
>My objection is the strange idea that humans are automatons that are keyed off input like a clockwork machine and operate sequentially. This is clearly not the case.
Well, a few thoughts here. First, it's worth noting that the argument isn't necessarily that AI are conscious in the way that humans are, nor that humans are strictly automatons.
But I think the more interesting thing is that our understanding about consciousness has evolved quite a bit in just the last fifty to one hundred years. We used to think that only humans were conscious, but assumed that primates, cows, dogs, and other mammals were just automatons. Then we started to think: okay, maybe primates are conscious. Then eventually: well, dogs also seem to have consciousness, and then rodents, etc.
This has continued such that most people in the study of consciousness think all mammals are conscious, and the debate is shifting down to insects and other creatures that we do think/have thought of more as automatons. We don't actually know where to draw the line, because it's essentially impossible to really feel/know the inner states of other living beings.
In the face of all this uncertainty, Chalmers just points out that since we understand consciousness so little, that ultimately we should probably be less definitive in pronouncing which things do or do not have it.
But the machine doesn't have to understand humans to do that. It gets trained on a whole bunch of sentences and then it is able to complete text. You could maybe claim that it "understands" the text but even that's a stretch.
Through training on human text, we are building implicitly in the weights a statistical model of what humans might write in response when presented with arbitrary pieces of text. It turns out that we can make these incredibly accurate.
If building an accurate internal model of something then using it to predict that thing’s behaviour is different to gaining understanding of that thing, we will need to pin down exactly what “understanding” means, or we are forever doomed to talk at cross purposes.
Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawberry might be broken up into two tokens 'straw' and 'berry'. It has trouble distinguishing features that are "sub-token" like specific letter sequences because it doesn't see letter sequences but just the token as a single atomic unit. 'Straw' and 'r' are two tokens but an LLM is entirely blind to the fact that 'straw' has one 'r' in it.
As an analogy, I might ask you to identify the relative activations of each of the three cone types on your retina as I present some solid color image to your eyes. But of course you can't do this, you simply do not have cognitive access to that information. Individual color experiences are your basic vision tokens.
The widespread mistake people keep making is assuming the development of intelligence in LLMs should follow the same trajectory that human intelligence takes as it develops into adult levels of intelligence. Thus deficiency in some capacity that we take for granted in humans is an indictment on LLM intelligence. But this is specious. LLMs are entirely alien; their developmental paths do not and should not look anything like ours. Your intuition from human intelligence just works against understanding the potential for intelligence in LLMs.
To be fair, almost everyone who claims LLMs are conscious tends to claim that they are conscious in exactly the way that humans are, to the point of stating that human brains are also just complex next-token prediction machines with a random seed. It's basically religious arguments on both sides.
I have seen people say "you're a next token prediction machine" but only in a similar way one might say "you're a cup of old lard". Not actually meaning it literally.
I have seen people interpret the request to show that they are not next token prediction machines to be a claim that they are, but this is almost always an argument to show certainty is difficult in this area.
People like Hinton have declared that they believe them to be conscious, but clealy indicate that they do not mean just like us.
And it seem obvious to me that language behavior does differ significantly between humans and LLMs based on the frequency and nature of failure states. LLMs routinely hallucinate, or get "AI strokes" or get obsessed about not talking about goblins, etc. This isn't typical language behavior for humans unless they have severe neurological or psychological impairment.
People tend not to "spew words out without thinking" and certainly not all the time by default - we call that glossolalia and (outside of some fringe Christian sects) it's considered a "bug" not a "feature" of the human brain. Human language by default always has intent behind it, even if that intent isn't readily apparent to the speaker. People can recite by rote memory, but that isn't blind token prediction, it's the neurological equivalent of muscle memory. People can have conversations then forget about them because their attention was focused elsewhere, but that doesn't indicate that they were simply "spewing words out without thinking" at the time.
People imagine details all the time. Eyewitness testimony is notoriously untrustworthy.
Our brains seem wired to confidently fill in gaps. We all have a literal blind spot we aren’t aware of because our brains convincingly lie to us and fill in the gap.
I don’t know what an “AI stroke” is, but I’ve definitely seen human beings in good health be in the middle of talking and suddenly forget what they are going to say.
> People tend not to "spew words out without thinking" and certainly not all the time by default - we call that glossolalia and (outside of some fringe Christian sects) it's considered a "bug" not a "feature" of the human brain.
Glossolalia is spouting gibberish, not comprehensible speech.
Kind of weird that you speak so confidently when you don’t apparently know the difference between steam of consciousness and “speaking in tongues”. Almost like you’re AI hallucinating.
This sounds like a description of a child who has not learned to read yet. You ask a child who is not aware of the alphabet and of "words" how many r's are in strawberry you'd get a non-sense answer too. So what you're really pointing out is that the LLMs have not been trained on "the english language" and how words are constructed and what they are composed of. That they operate by tokens that don't correspond to words or letters is irrelevant as an answer to why they can't count the letters in a word. It's not that I know how many r's are in strawberry because of how I'm understanding the word "strawberry", I know how many r's are in strawberry because I know how to spell strawberry. The LLM needs to be trained on this the same way someone who is learning to read would be trained on it. No one should be surprised that an LLM can't "read" in the same way no one should be surprised that a child can't "read".
This interpretation takes things too far away from how LLMs are constituted and so misses important explanatory power. The issue of counting letters in a word isn't about an ability to spell, it's about the nature of one's perception. We perceive words as sequences of individual letters. LLMs do not. I can ask you to tell me how many r's are in some nonsense word sequence and you're fully capable of doing that. LLMs do not see sequences of letters so they are intrinsically at a disadvantage for this kind of question. But this says nothing about its capacity for intelligence anymore than not naturally being able to distinguish frequencies of photons hitting your retina has anything to say about human intelligence.
Counting letters in a word seems to have little to do with understanding the word. Young kids can’t spell or count well at all but no one says that means they can’t understand.
This is where the other claim is being made. That the structure of the model is fundamentally incapable of the operation, so even if you stipulated that the way you provide data is sufficient for intelligence then it still wouldn't work.
The universal approximation theorem addresses this point. In that, with an identity attention mechanism, a LLM is just a multi layer perceptron. The attention mechanism is effectively a way to get one of the benefits of a much larger fully connected layer without the massive cost.
A LLM can do what a MLP can do. A large enough MLP can do any function to arbitrary precision.
That makes the claim that an LLM could not do a task the same as saying no function can do that task.
Some are ok with this, if you invoke some supernatual aspect to intelligence then the inability to describe it with a function is quite reasonable,
If you want to stay in the world of reality, you have a much harder task, people like to point at quantum (Penrose) but it's hard to say what it is you are pointing at.
I think the very act of proving that something is or is not intelligent, would render it functional by nature of it having a proof, (or disprove Gödel's incompleteness (a tough ask))
Are there any proofs that cannot be expressed as a function? A kind of Gödel locator, where you can prove something that you can identify is true but there is no formula to express it. I'm not entirely sure what that would even mean,
A language model completes text based on the overlapping patterns of the training data.
There absolutely was thinking involved… in the training data. Same as when you read a book, you engage with the thinking behind the text. The book isn’t thinking, and the author may be dead and gone, but there’s absolutely the traces of thinking in the text.
Language models produce mashups of texts they were trained on, and there’s absolutely the traces of thoughts behind those mashups.
I'm hearing a lot of bad arguments against LLM consciousness lately. Bad argumentation heralds bad outcomes.
What bad outcomes do you foresee from badly arguing against LLM consciousness?
We discovered math that decodes data storage in langauge and is able to use sophisticated continuation cohorts from ALL OF HUMAN RECORDED KNOWLEDGE to respond to you in a call/response model with very good synthesis capabilities.
Its super useful, but not life or conciousness. Its a simulated echo from our collective recorded behaviors. It understands because we understood first. It replies because we wrote it first. And it sorts, organizes, synthesizes and compresses that at impressive speed now.
It’s strange many others have not eh? I think when new developments arise, ironically, this is the true measure of human intelligence - one’s ability to make sense of a thing and be closest to the truth.
From Wikipedia: In 2023, Chiang was named one of Time's 100 most influential people in AI.
Indeed it isn't a one-off. His last infamous article compared AIs to Xerox machine image compression. He convinces a certain type of crowd that is not technical enough to poke holes in his posturing.