the subject is consciousness, not intelligence.
How can you have a subjective experience without a body?
That's the point, in my opinion: your physical/chemical state (body) in a given moment is then translated into the higher abstraction of the emotion. An emotion that *you* feel, because you are self aware of what's happening.
How can you be self aware without feeling? And how do you feel, without a body?
Easy, you don't have subjective experiences because you have body in the first place. You have them because some signals come in from your nerves, which your brain turns into a world model. You are effectively a "brain in a vat", the vat just happens to be placed on top of your body.
An AI system constructs the world model a little different, by all the text that gets feed into it, but that doesn't mean that there is anything fundamentally different in the world model it builds. Consciousness operates on world model, not on the world or even the body itself.
The AI's world model might be missing some information, because they weren't described in enough detail in text, but that shouldn't matter for consciousness. A blind or deaf person isn't less conscious than one that can see or hear just because some information is missing from their world model.
If we want to know if AI is conscious or not, we have to ask if the AI can recognize itself in the input it gets.
Some aspects like limited content length and lack of ability for the model weights to update will certainly limit what the AI can do. But that's ultimately a matter of degree, not kind, when it comes to consciousness.
It is important, if I destroy your brain and grow a new one and start running the same program on the new brain your consciousness is still dead and its a new person living.
Computer AI models don't run on a single machine, they run in a distributed manner using different machines at different times. When you ask a follow up question thats not sent to the same machine, its another machine answering. So the consciousness one of those computing machines experience would be extremely fragmented and not at all conscious about any discussion with you, since it only saw a few fragments of it.
And no consciousness doesn't expand to cover larger distributed computations, otherwise social media would have developed a consciousness by now but it hasn't. Groups of humans don't start to share consciousness, it doesn't happen, so you can assume groups of distributed computers wont as well.
If I copy a text document from one computer to another, is that the same document or a different one? It's all just information. If you copy it, you have two, it's still the same until the documents start changing and go different directions.
> Computer AI models don't run on a single machine
It doesn't matter on how many machines it runs on. It's information processing, as long as it gives the same results, it doesn't matter how you accomplish it.
Consciousness isn't some magic thing that sits on top, it's the result of that information processing. You take random sensory data, the brain transforms that into "cats, dogs, you, me", it uses uses those percepts to execute actions, gets more data back and checks how the actions changed the world state.
Keeping track of what changes in the world were causes by actions of the brain vs things that happened due to other causes is the conscious experience.
If a hundred people see the same event, will they all respond the same?
Pretty sure you wont, humans vary the exact wording. They will say the same song but they wont answer the exact same way every time. Even if they say the same words two times they wont use the same tone and body language, as they don't just communicate via words and that nonverbal language is a part of what we say.
Nicely circling back to LLMs not being able to learn and form memories.
If you have hundred different people, they will of course do something different. Just like hundred different AI model will do something different. The question you have to ask is if the same person under the same circumstances would do the same.
Luckily, we have an answer to that: They would. Transient Global Amnesia is a condition where people temporarily lose the ability to form memories and in turn they keep repeating the same conversation again and again[1]. Their brain keeps asking the same question again and again, as it doesn't remember the answers it already got.
Bro, I think we discarded this idea from Platone and Cartesio a while ago...
Your brain is your body.
Your mind is not detached from it, and you can't feel anything, and so have a subjective experience, without it. Neither, your mind, or "soul" could survive to the physical death of your body.
So...I mean...
You don't need a body, you need electrical signals your brain interprets as body. And in principle you don't even need a brain, you could replace that with some matrix multiplication or transistors that do the same stuff.
The important part of consciousness is being able to figure out what of the sensory input is correlated to your own action and which was caused by the rest of the world.
Then a request comes in, and the system does a bunch of calculations using those bits, and spits out a result. The bits are unchanged.
When your brain receives input, it is changed. It is constantly active. If it ever stops being active it's dead.
So, what exactly is the claim? Are the bits constantly conscious? Do they snap into consciousness when the computer does math with them? Or is it maybe the computer that's conscious while it's processing these bits? How about when it stops doing that and goes back to doing other stuff? Why are these particular bits special? Was the computer always conscious?
I feel like the only way anyone could believe LLMs are conscious is if they don't understand how computers work. Of course it isn't conscious, how could it possibly be conscious? Its literally just bits. It's like saying the text in a book is conscious.
These are all fine questions, and they don't become any easier to answer if you replace "computers" with "brains" and "bits" with "neurons".
Of course it isn't conscious, how could it possibly be conscious? Its literally just bits.
"That's ridiculous. How can meat make a machine? You're asking me to believe in sentient meat."
What is even being argued here - neuroscience is hard, so programming your PC thus makes it conscious?
> "That's ridiculous. How can meat make a machine? You're asking me to believe in sentient meat."
… what?
We don't understand how combining a bunch of obviously(?) non-conscious biological components can produce a larger system that is conscious, so it's unwarranted to be certain that that can't happen with software.
Or if they're retards. The fact this still comes up is weird. A printing press isn't conscious, so why would an LLM be.
Don't forget, some of the bros are overly excitable. Like that twat who reckoned a Google model 5 years ago was conscious.
Why can't that physical medium be GPUs and RAM? And temperature sensors and cameras? What's special about our meat that it's our "body" in the way a computer is not the body of an AI?
I don't think the point being argued can be true without some incredibly contrived, human centric definitions of "body".
Would you be "you" in a different body?
To claim otherwise would mean anyone who's gotten a transplant or amputation is no longer themselves.
If you delete the hardware and buy new hardware and write in the same weights its still the same LLM. If you delete your brain, get a new brain and put in the same weights in the neurons its a clone, its not you. We know what happens when we create a clone, the clones aren't the same person they are separate consciousness, so consciousness is tied to the hardware.
For example, if we have 5 identical clones down to the atoms of their brains, and send a message to the first and let him respond, then send the same message to the second together with the firsts response and let them add a new message etc. That is not one consciousness responding, thats 5 different consciousnesses responding, and that process doesn't connect them into a single consciousness. That is how LLM works, at best a single token generation is conscious, but that would be a less meaningful consciousness than a ringworm.
Your whole argument hinges on this, and I don't think it's true.
A clone is not what you describe, not even close. Clones are copies of DNA, not of the actual physical neuron structures.
If you were to vanish, and the next moment a perfect atom-for-atom, particle-for-particle, quark-for-quark copy appear in exactly the same place, I think that is you. As far as I know, that does happen every single planck second.
For it to be you, though, it needs to occupy the same space you were previously in. You can't copy your atoms 5 times over in 5 different locations; those aren't you, they're just similar. Like copying an LLM 5 times over to different hardware, seeding its RNG differently each time.
The experience of “you” is just your specific memories and world model, continuously updated with sensory input.
If another body “runs” the same exact pattern, that is you. Theres no link and nothing was transferred; the pattern of thoughts and memories is all “you” ever was.
Same as playing the same song on two different speakers. Nobody asks what links the song across them; it’s the same song wherever it’s played. You’re just a far more complicated pattern on a far more complicated speaker.
You might ask, “but why am I this pattern?” Because this is the specific pattern modeling itself from the inside in asking that question.
And my intuition is that no, you can’t, they are two aspects of the same reality.
If "I" am in a different body, then what makes that "me" who's in a different body?
I don't see why hardware is any more fungible than a kidney. If your LLM reads the serial numbers of its motherboard/RAM/etc as a seed for entropy you can make identical arguments about body fungibility and self.
If every neuron of your brain were simulated precisely on a sufficiently powerful computer, that simulation would have subjective experiences, without having a body.
Also that's impossible. It is impossible to simulate reality exactly using digital computers. The best we can do is approximate. Doesn't matter how powerful it gets, it'll always just be an approximation.
It is nonsensical to claim that anything other than my brain could produce the same consciousness that my brain is producing. It's obviously far beyond anything any conceivable digital computer could ever reasonably simulate, and even if you did create a "good" simulation it obviously wouldn't have the same properties that my brain does because it's an entirely different thing than my brain is.
In any case, a closely-but-not-perfectly-accurate simulation of a real human brain is still going to be human, unless you believe that someone becomes less human when they're experiencing some kind of cognitive decline, or a stroke, or other biological malfunctions. The point is, there is nothing essential to the having of a physical brain that creates the concept of consciousness or sense-of-self.
And there absolutely is something essential to the concept of being human that we are entirely incapable of replicating artificially. In fact as far as I know we are incapable of synthesizing any kind of life whatsoever. We can't even create the simplest type of living cell imaginable.
So to claim that we could just create consciousness, a fundamental property of this "life" thing that we don't properly understand, within a piece of rock is beyond naive. We don't even know what it is or how it is.
We understand quite well where in the brain the sensation of self-awareness / self-experience / sense-of-self comes from. We have evidence that disruption of that part of the brain breaks those sensations.
Okay now tell me where in the LLM its alleged consciousness comes from.
Great q. Deepening it further-how can you have a subjective experience without consciousness, which isn't necessarily tied to physicality. Taking it one step further-can you have consciousness without a mind? Who's the first mind, the first cause of it all, that begot both the material and immaterial world?
Fun stuff eh?
Some anecdotal data.
Many dreams I have are just of the computer screen of some coding problem. I think the problem could be x, so I try x. But I don't type the keyboard or anything, the code just magically appears as soon as I think of the solution. Then run the code (but no clicking) and it works or not. I feel in the dream success feeling or failure feeling but there is no body at all.
Also I have other dreams where there is no body that I am aware of but not going there in public.
There is no body sensation in these dreams. But dreaming is very much being consciousness as well as feeling emotions. So answering your question its possible to have a subjective experience without a body but whether you needed the body to learn to have that sensation without a body in the first place is unanswered.
I suspect sensory inputs are more important than a body. If that is the case then eyes can be replaced with cameras, ears with microphones etc. Text input is just another sensory input.
If I hooked up electrodes to the hearing centers of your brain and force fed you dialog you perceive as speech (but is really a great deceiver), then responded in what you thought was speech (but are really just probes I use to convert your thoughts to text), that wouldn't suddenly be less real to you. It wouldn't devalue your sapience.
Embodied cognition rejects this assumption. We didn't evolve as brains that were then put in bodies, we evolved as bodies with nervous systems.
So I have no idea what distinction Ted Chiang is trying to draw
The point here is not that it must have a body like ours, the point is that a conscious entity must have a boundary line between internal (the body) and external (everything else).
A virtual sense organ can simply be an encoder or a web camera or a magnetometer, the specifics don't matter, what matters is that there are only a few bridges between the outer world and the inner world.
Even if you want to call a tokenizer and autoencoder a "sense organ", LLMs are not embodied because there is no boundary line - there is no internal "thought" that is not directly descended from the prompt and there is no internal reasoning which is not immediately dumped into the external environment.
Related, is a human "thinking out loud" still thinking, even though the internal reasoning is "immediately dumped into the external environment"?
No, you’re not. I think even baseline emotional responses to stimuli is table stakes for “consciousness”.
Genuine question: How are you so sure whether you are experiencing an emotion or not? Are you a master of everything inside your own head?
Damn, what a line!
Another thing that bothered me with his baseline for consciousness was that it did not involve the ability to change one's self. A big part of being conscious in my mind is how one's experiences shape them, and how someone can shape themselves. LLMs completely lack this, their weights are static. An LLM isn't going to be molded by a bad breakup, or a relative passing away. An LLM isn't going to set up a routine to get stronger with training, nor smarter by reading up on a field.
We humans have mostly frozen weights (neurons), or else we would constantly be having to avoid forgetting how to walk+talk. We have a period of greater plasticity (youth!), and use sleep and dreams to perform 'deeper' updates than occur when we're awake: We tend to suck a bit at picking up new skills from zero, but improve rapidly with practice over days.
Input stream comes in, input stream comes out. The LLM doesn't care whether this happens once a minute or once a year.
Nothing prevents one from running an LLM whose harness has a clock and a while loop in it, and it would be weird if its mere lack were really so consequential to consciousness.
Your brain and your whole body exist in time. Even when you are asleep, your body does not flicker out of existence and your brain actually continues working during that time.