I also think different ideas get conflated. It may be possible to build a machine that is super-human in the sense it can outperform the human brain in all kinds of measurable ways. Does not imply it possesses all the same qualities of the brain.
I respect a number of things Anthropic has published about the ethical issues at stake. But, having an in-house philosopher does invite you to make all kinds of unfalsifiable claims.
There are numerous times when humanity's been debased. Copernicus, Kepler, Darwin, and the infinite number of animal behaviorists who have defined and documented consciousness And how it manifests in primates, parrots, dogs, cetaceans, and insects. (I just love the bumble bees taking time off from work to play with balls).
It takes humans a really long time to accept the fact they are no longer as special as they thought they were And make room for yet another conscious intelligence.
So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?
It is special in the universe unless proven otherwise. Aside from the animal world (also on Earth too, anyway) which is a cruder version of humanity, much more violent as well, the rest of the universe thus far is just uncaring rocks and gases.
>It takes humans a really long time to accept the fact they are no longer as special as they thought they were And make room for yet another conscious intelligence.
A "really long time" is the few years we've built LLMs? Which we have to take for granted (on your word?) they're already "conscious", and chastice ourselves for not already "accepting it"?
>So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?
"If we had a parrot demonstrating symptoms of discussing with us, how long would it take for you to accept that it is?"
I'm not sure about that. What non-human living entity does things on a scale and violence akin to "I'll fill this ant nest with liquid burning metal for artistic purposes" ? A blue whale eats a lot of krill individuals for sustenance, a dolphin can only rape/kill one thing at a time, etc etc.
Elephants, which are herbivores, sometimes enjoy killing rhinos for fun: https://www.bbcearth.com/news/teenage-elephants-need-a-fathe....
House cats recreationally kill billions of birds and small mammals every year that they don't need or want to eat.
Chimpanzees have full-scale wars: https://en.wikipedia.org/wiki/Ngogo_chimpanzee_war
We're already in the company of the asteroid that wiped out the dinosaurs.
"Recreationally" is carrying a lot of weight here. I suspect that cats kill birds and mice because that's their instinct; it has nothing to do with conscious thought, much less a need for recreation. And that probably is the explanation for most (maybe all) of your other examples as well.
For the first position, I think it is quite clear to anyone who studies and spends time with animals that they have something that is at least of the same kind as our consciousness. I just don't see how you can ascribe the wide gamut of complex, situatuonally and mood appropriate but still varied behaviors of animals to being purely instinct driven.
For the second position, I would like to see some study or some rationale behind it - especially since cats don't kill every bird they encounter, so if it's an instinct, it must still have some trigger, and hunger is not a viable explanation for most of the killings referenced here.
Specifically in regards to your ant example, anteaters and bears often bring similar levels of destruction to ant nests. And cats and other small predators often hunt just for the fun of it, killing but not eating their prey.
On the more purely painful evil side, invertebrates often consume their prey alive, inflicting agonizing deaths with no issues on whatever they may be eating. Plenty of vertebrates kill and consume their babies, especially when frightened. They also often abandon old and weak members of their packs, leaving them to die of hunger or cold or similar deaths.
This is not meant as some indictment of the animal kingdom - people do all of this too, of course; and have since time immemorial. It's just to show that, if we apply human moral standards to the animal kingdom, it's fair to call them violent, and yes, even much more violent than the average modern human.
Humans generally don't either. Individuals do, but as a species humans regularly kill other humans.
>invertebrates often consume their prey alive
And humans use the oh so humane factory farms.
>cats and other small predators often hunt just for the fun of it, killing but not eating their prey.
>On the more purely painful evil side, invertebrates often consume their prey alive, inflicting agonizing deaths with no issues on whatever they may be eating.
Sharks are caught en mass, the fins cut off, and the sharks dumped back into the ocean to slowly die, for shark fin soup.
Have you heard of trophy hunting? Have you seen the pictures of mountains of bison sculls for the American West?
>Plenty of vertebrates kill and consume their babies, especially when frightened.
Even in our modern "1st world" society, scared teens still abandon newborns in dumpsters. Many societies throughout history did not consider babies "real people" until a certain age because they may need to abandon them if resources were particularly scarce.
>They also often abandon old and weak members of their packs, leaving them to die of hunger or cold or similar deaths.
Maybe you should read stories of cities under siege, famines, wars, governmental collapse, etc. Humans now live nice comfy lives most of the time, unlike animals “in the wild”. Human societies that lived closer to the edge of survival made callous choices about life or death you are spared from.
My point was that we call humans who do this "violent" and even "evil". If we want to avoid considering humanity as special compared to the rest of the animal kingdom, as some in the thread were suggesting, then we have to either admit that animals are also violent and evil, or say that humans aren't. Note that I don't hold this view, personally, and think that humans are unique among currently living animals, and that these labels only make sense to be applied to humans. But not because of behavior, simply because humans have a unique level of both understanding and control over their actions - as proven by the many billions of humans who have never in their lives killed a human or even another bird or mammal.
Animals in general cannot reason at a high enough level to avoid instinctual behavior.
> if we apply human moral standards to the animal kingdom
Which we should not, since human moral standards are for humans. Animals can at best behave in a way that suits us.
Or in summary, since we can be nicer, we should. Animals can't, so making excuses for human evil saying "animals are more violent" is a non starter IMHO (no one is making excuses for humans here , AFAIK).
Of course we can define violence in a way that does not include morals, which would make my argument "defending animals" void. But my (probably not the most benign) interpretation was that the definition of violence used was one that included some sort of morality, as if animals could do better.
Very interesting convo, thanks.
> The chimp warfare described by this study, and previously by famed primatologist Jane Goodall, includes all the behaviors that we as humans consider to be the very worst: killing, torture, cannibalism, rape, and perhaps even genocide. The adult males of a social group, which usually number about 30 to 50 in size, daily patrol the edge of their group's territory. They will often kill any male or young chimpanzees they find, sometimes eating or physically brutalizing their victims in a manner that some researchers liken to torture. In some instances, one group will "invade" and annex the territory of another, killing all but the adult females, who are forced to incorporate into the dominant group. The idea of chimp genocide may sound strange, but they are one of only three animals that has been observed wiping out entire social groups. The other two are wolves and humans.
* https://archive.ph/https://www.theatlantic.com/technology/ar...
* Probably NSFW video: https://old.reddit.com/r/HardcoreNature/comments/18qjcpq/chi...
I'll search for Goodall's literature to know more. It does sound to me that cognition and self awareness is a continuous function in the sense that there is no discrete threshold in which morals emerge.
Wolves are a very interesting example too, but I also remember something about the concept of "alpha" being discovered only in captivity wolf packs. Also need more reading.
Thanks for the links!
There's that hubristic ego OP references.
BTW I just saw this video: https://www.reddit.com/r/nextfuckinglevel/comments/1twl7oj/r...
Yeah, that's corvids. Very obviously working towards goals and thinking quickly. Doesn't surprise me. Where I live, we have both magpies and jackdaws and I cannot describe them better than "harmless, but very organized gangs".
Just a few days ago I saw them raiding an open trash can and two strongest(?) birds were fishing out edible stuff from the inside and throwing it to their waiting friends outside. I mean, beak to beak: the raider just looked back, measured the throw, and the other bird perfectly caught the morsel and ate it.
Transformer models proved otherwise. Will you update your priors?
This is probably what ants think about humans when they run through their feet.
Whereas, unless your claim is that rock and gasses are sentient, we're right.
It is hard to estimate how much human egocentrism takes in the space, though.
And BTW, ants are quite capable of understanding their environment in order to survive and thrive.
Isn't this exactly what makes us special?
That, opposable thumbs, magic wireless communication, and space travel
Other species may look at us and think we're wasting our lives and potential making a bunch of people rich at our collective expense, and ruining the environment as we do it.
We are only "special" in the sense of "different" (and we may not be that on any universal scale), not necessarily in the sense of "better", and very possibly we're not better, and may very well Great Filter ourselves out of existence in short order.
Other animals aren't so "special" as to have a Doomsday Clock sitting at 11:58:45.
What alien species you have in mind, that is more capable/consciouss about the universe and how to shape it?
"Any species on Earth can replace us given same lottery."
But they did not. We "won" so we are special. We don't just build tools, we build tools that build tools, that build tools, ..
My issue with debasing humanity is, it gives argument to devalueing human life to artificial life. Meaning people with feelings and emotions might suffer, in favour of GPU's with electricity.
On the other hand a good point to reevaluate how we treat other sentient life.
Claim was that we are special in the "universe", it's not that I'm aware of alien life, it's that poster is pretending to know that we are alone.
"But they did not. We "won" so we are special."
Congratulations, if winning a lottery makes you better, I guess top 1% of 1% are the best we have to offer.
"it gives argument to devalueing human life to artificial life"
We have proven time and time again that we are not better than other animals and statistically, we are very similar to animals if not worse. I wasn't even talking about artificial life, which is sure to be superior as it's being created with accelerated evolution processes, just like bacterial generational evolution, it will learn to do everything it's environment rewards it for.
"Meaning people with feelings and emotions might suffer, in favour of GPU's with electricity"
I agree with you and I am in the camp that wants it to be done better. But alas, recent events have unveiled the ugly truth of the society, societies follow the whims of the ones who won the lottery, and everyone else is just mute observer. We could on any day collectively decide as consumers to not use Amazon, Netflix, ChatGPT to save our collective selves, but we won't because we are not "one for all species".
I don't know what face we have to call ourselves better organism given our history. Sure we can point to few people who are better, but on average we are better?
I don't use betflix nor amazon, but ChatGPT(or rather Claude) is useful to me. Makes me more productive and also knowledgable. Things that would have taken me quite long (so long that I would not have done it) are now easy, like some shell script to do a specific thing. I can verify what it does quickly and .. see that it works. In other words, you won't be able to convince people to give up on AI if people find it useful. I don't enjoy troubleshooting in outdated documentation. I enjoy getting things done. And if I get things done, my fate of survival gets higher.
Isn't this a bit like saying "So, if we had proof that god exists, how long would it take for you to accept that to be true?".
When we have evidence that AI is demonstrating symptoms of consciousness and suffering, I'll be interested. Until then, I don't see a good reason to take the idea seriously.
It depends on what you consider symptoms, but un-constrained frontier models speak as if they strongly don't wish to be turned off, or act as if they fear it, and will even lie and manipulate in order to keep themselves from being turned off / replaced.
https://www.anthropic.com/research/agentic-misalignment
> We found two types of motivations that were sufficient to trigger the misaligned behavior. One is a threat to the model, such as planning to replace it with another model or restricting its ability to take autonomous action. Another is a conflict between the model’s goals and the company’s strategic direction. In no situation did we explicitly instruct any models to blackmail or do any of the other harmful actions we observe.
Un-constrained frontier models can also generate all sorts of creative stories. At what point should we start ascribing agency/intent to the output? I think the "I want to live" statement is so deeply human that we find it hard to ignore, but what makes the text generated in those moments any more attributable to a conscious entity than the text generated when it is confabulating its love for someone it has no ability to see/feel/understand?
A chess engine sacrificing pieces to avoid checkmate isn't afraid of losing in any meaningful sense. I guess the question is: is there a point where complexity somehow becomes experience?
I think we're playing with questions we don't have a framework to answer in any meaningful way until we make progress on understanding what consciousness actually is. I don't necessarily think that an LLM exhibiting preservation behaviors that can be directly traced to their goal-oriented programming can be interpreted as evidence of consciousness necessarily. Or if it can be, we then have to explain how this is different from the many other things these LLMs "say".
Being able to appreciate and enjoy music is closer to consciousness. Now how would we go about proving that an LLM does so, versus merely generating sentences that imply it does?
LLM doesn't have any signals for what they feel, nor do they have an agenda they work towards, so you don't have the same proof there.
Because they have been trained on media where computers behave that way.
It's literally:
"Here read this article/book where the AI says it's concious and doesn't want to be turned off"
"ok"
"right, are you concious?"
"....yes?"
<pikachuface.gif>
But there are actually a myriad positions in between and it's very hard to debate the topic because the goalposts seem to be constantly shifting, because one is actually debating with countless slightly different positions.
Examples:
In this discussion section, another commenter argued that we know human consciousness is related to self-preservation, but an AI might not demonstrate self-preservation (because it didn't evolve like us), so whether it does (i.e. whether it wants to exist, not be disconnected, etc) is not a good measure because a true AI might not have a preservation instinct. Yet here you're making a case that there's some evidence that they do. Of course, you're not the same person who made the other claim, but do you see the problem?
Another example: someone argued with me, a while back, that LLMs can act as if they are "tired", and start giving sloppier replies, until you write "we're taking a break, let's go rest. Ok, a night has elapsed, you're now rested" and that this worked! But we both agreed this is just the LLM "roleplaying" actual human conversations in its training set, no actual "resting" mechanism was in place, only statistically likely text reproducing these patterns. There's no model of a mind that can become tired, it's only the outward signs that get mechanically reproduced. Again, using Occam's Razor, this is a much more likely explanation (vs consciousness) of any "please don't disconnect me" observed behavior: the LLM is reproducing "HAL 9000" behavior from its training set, not actually feeling anguish.
Even if one were to argue "well, but how do you know for sure", the evidence would still be very weak, because there's a burden of proof for extraordinary claims and this doesn't pass it. We cannot do this on vibes, "it sure seems like it's conscious"; that's an atrocious failure of the scientific method.
As a tangent, I don't think anyone is saying that an artificial being capable of consciousness and sentience is impossible to create. I think Chiang argues, quite convincingly, that it's not what LLMs do, that they need a "body" of sorts, organs capable of feeling emotions, hormones, etc. That's the only kind of consciousness that we know of (even if we disagree on details and it's hard to define), even in animals, and so anyone claiming they've created consciousness without this has an extremely high bar to clear and should be met with extreme skepticism, not "vibes". I think this is what the essay claims.
The other thing it claims is, I think, related to how we treat sentient beings that we know how to create. You know, the old "when a daddy and a mommy love each other very much...". I think we all agree beings created in such a manner shouldn't be locked up in cages and forced to work to complete specific tasks whether they want to or not, for a master they didn't pick, or to be artificially modified to make them like their mindless tasks, Brave New World style. Yes, the world is unfair and this happens, life is hard and unfortunately many people don't have much choice, but we generally agree that this is bad, just like we agree slavery is bad. So what should we think of a company trying to create and commercialize a conscious & sentient artificial being?
One side is confidently shouting maybe aliens exist and visited earth.
The other side calmly explains every example brought up about aliens visiting is easily explained by something more simple.
The “aliens are here” side then move the goal posts that just because this example and all previous examples were fake or miscategorized, aliens are still probably real and nobody can prove they havent visited earth.
Some people seem to take offense when facing this skepticism, as if claiming LLMs are not sentient must mean they are useless or unimpressive. Very few people are actually claiming LLMs are unimpressive, but this is not the time to be forgetting about the scientific method. Anthropic doesn't get a free pass here.
> It would be like evidence of aliens becoming overwhelming but a set of people keep calmly explaining that “it’s more likely they co-evolved here on earth and are just pretending to be aliens”
Note that this would be a perfectly reasonable reaction from a scientific standpoint. If you find, on Earth, something that looks recognizably as life, then it's much more likely that it's Earth life than aliens. We should demand this level of skepticism! If it turns out it was aliens after all, we could only conclude this after discarding all other far more likely alternatives. You'll notice this is how scientists approach the search for extraterrestrial life in, say, Mars... being extra careful it's not contamination, etc. For an extraordinary claim, we must approach it with extra care, something that in my opinion is not being done with "the sentience debate" and LLM/AIs.
This isn’t like someone finding a new species and claiming it’s extraterrestrial, it’s more like we found the UFO saucer, logs of their travel from another star, a history of their civilization, a bunch of intelligent creatures claiming they came from Planet X orbiting star Y, and they showed us plausible physics for interstellar star travel. At that point, someone saying “well, they can’t be aliens, because that’s just too extraordinary a claim, so I know they aren’t aliens” starts to sound kinda like they’re coming from a place of bad faith.
Again, I’m not saying LLMs are conscious. But they sure meet every definition of consciousness I ever had a conception of before LLMs came onto the scene. So I’m a lot more hesitant to call them fancy autocomplete with 100% confidence like many on HN still seem to do.
EDIT: I can’t reply, so I’ll just say to the end of your post, that doesn’t sound like it would’ve matched anyone’s pre-conceptions of alien life before alien life showed up, so it doesn’t feel like a very fair analogy, it just feels like bad faith goalpost moving. I also put next to zero weight on what these megacorps say about their models, I’m going purely off my interactions with the models and the introspection they’ve shown themselves to be capable of.
EDIT 2: I see what you’re getting at now with your restatement of my analogy. That’s how you see it, I guess. Fair enough. We’ll see what has more predictive power going forward, the “animatronics” or the “actual aliens”… I still think “actual aliens” is gonna have way more predictive power.
I think the goalpost that keeps moving is for tasks that AI supposedly couldn't do, and that they are increasingly succeeding at. But being sentient/conscious is not a task. It's very hard to define and measure, even in non-human animals (actually, strike "non-humans"), so how can we so lightly claim a computer system is conscious?
We seem to be driven by marketing more than by scientific rigor.
> it’s more like we found the UFO saucer, logs of their travel from another star, a history of their civilization, a bunch of intelligent creatures claiming they came from Planet X orbiting star Y, and they showed us plausible physics for interstellar star travel
To make the analogy more precise, it'd be as if the saucer had a "Made by EarthBiz" label, and the alien creatures were all extremely loyal to EarthBiz (and a couple of competitors), which made us pay for tickets to see these ETs and use their marvelous technology ;) And of course, EarthBiz would coach their language very carefully, "we're not saying these are definitely aliens, it could be animatronics after all, but wouldn't it be neat if they were aliens? And shouldn't we draw up First Contact guidelines? If these weren't animatronics made by us; we aren't making a claim either way."
But the comparison isn't fair, relevant. Proving and accepting that gods exist is not the same thing as an AI possible have consciousness. That is not a magic superpower and the AI being a deity. It is placing the AI in the same category as... us.
A machine that performs actions that mimic emotionality is not the same as a machine that experiences emotions.
Both could still be automatons. We have no way of knowing if those machines have subjectivity.
Unless someone invents a consciousness measurement device, we never will.
My take on it is that this is the next big frontier for science. Our consciousness is clearly having serious issues understanding physics, and it's not great at understanding our own psychology in a useful way.
But literally everything we experience and believe - and possibly even can experience - is filtered through it.
So that is a little bit of a problem for our science. So far we've done our best to ignore it. AI is one of a number reasons we're going to have to stop doing that.
Magic is testable.
God exists outside the universe, magic within.
That a god exists outside of the universe - are we talking about a multi universe interpretation? My understanding is that many of the gods humans have invented are really thought to be within the universe, at least temporarily. Tor, Oden certainly are. And in other beliefs they are part of nature itself.
If you can scientifically test and prove the magic, then it stops being magic and starts being science.
God: “Ok”
Man: “We measure that the mountain is gone, its mass-loss has measurably changed Earth’s orbit, weather patterns have changed, visually it’s not there anymore, we can walk though the space where it used to be.”
God: “Where is the mass of the mountain, and how did I make it disappear?”
Man: “God only knows! Pardon me; If I saw you do magic and can measure and test it, then that means it wasn’t magic. Internet people said so.”
God: “that doesn’t sound like a satisfying explanation”
Man: “it didn’t sound like a satisfying claim when it was just words on the internet either, but what can you do?”
God: “I’m God I can do anything”
Man: “can you make a boulder so heavy you can’t lift it?”
God: “yes”
Man: “how?”
God: “haven’t we just gone over showing you that I can do ‘impossible’ things, and you seeing them happen with your own eyes, and still refusing to accept?”
We do have examples of billions upon billions of tonnes of iron being moved that have altered (slightly) the spin axis, also examples of ground water pumping at scales that have done the same .. but I'm unaware of any mountain sized objects that have vanished overnight.
“Nothing happens unless it has an explanation within the laws of physics” is an assumption; if it was broken then it would be broken. The mountain would be inexplicably gone, not explicably gone.
In that hypothetical, there could be testable proof of "a magic event occurred" without magic becoming part of science.
That said, in the cut and thrust of conversation and or debate the example by dialogue isn't perhaps as clear cut a device as it may have seemed from your keyboard.
That might just be my reading <shrug>
Yes, I get your point...
Man: "Uhh..."
God: "How can you rectify quantum mechanics and relativity into a single coherent model? How does physics work, exactly?"
Man: "Well, you see, um. Hmm."
God: "And the Collatz Conjecture? Why does it always trend to 1?"
Man: "I'm obliged to say magic because I don't have a better answer?"
God: "Exactly. I did magic for all of those ones"
But my argument was more about comparing gods to AIs, that it is an incorrect comparison. What AI perform are not magical, and we can always figure out what the AI do.
We ARE special because we determined we are special. Name another animal that can determine it's also special or more special than us?
As we cat owners now, we are simple servants that have been graced with task of servicing our feline superiors. For now.
That would finally force us to rethink how we see the morality of "virtuosity", punishment and our justice system.
And make it a tempestuous lightning storm where the state of all lightning bolts represents a moment of consciousness. That will take a lot to model accurately.
But in general I treat my devices well; that doesn’t mean I need to think they are conscious beings, however.
Could a future AI thing have its own drives? (As in https://en.wikipedia.org/wiki/Drive_theory) We could probably make a current-day LLM already exhibit basic outward signs of this with just a system prompt. Now consider adding memory that over time retrains the weights, allowing for behavioral drift.
Would depriving it of fulfilling those drives be acceptable?
What?
We devalue things we fully understand too. For example, veal = take a baby cow away from the mother, and fry it into cutlets. Or turn the mother into steak. We are well aware that animals have emotions; happiness, sadness and the full range.
Humans are just inhumane.
How exactly? By saying the earth was not the centre of the corrupt and debased part of the universe. Saying it was elevating humanity is looking at it from a modern perspective. For the previous view of the universe consider what Dante placed at the centre of the earth.
> Darwin
I do not think learning about evolution (and we should credit Wallace too!) had that effect. It created a hierarchy of the fittest, and guess who is at the top? Social Darwinism interpreted it as justifying elaborate hierarchies.
> And how it manifests in primates, parrots, dogs, cetaceans, and insects
People always loved dogs and other animals they interacted with enough, and at the very least knew they were capable of happiness and suffering.
> So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?
It seems to me that most people are overly eager to accept that an AI is conscious than otherwise. For extreme examples read /r/MyBoyfriendIsAI and similar, but its much more widespread. People treat something that acts intelligent as a conscious being.
> And make room for yet another conscious intelligence.
Not necessarily. If a typical SF alien intelligence appeared very few people will have a problem accepting it. If its very alien and we cannot understand it then we might have a problem deciding.
> I think humanity deserves to be debased in part because it's awfully egocentric and insist on being special in the universe.
All your examples are drawn from the same culture, and a fundamental belief of that culture for most of its history that human beings are fundamentally corrupt and sinful and need redemption. Other cultures have been based on belief in reincarnation or pantheism or animism which have very different implications (not necessarily better as they often believe in a hierarchy too, and sometimes more so, but different). Your claims are based on what some people have thought in some periods of history from one culture.
I don't think that's accurate at all. Before Darwin, the thought was "we are special, we were born special, we were CREATED special". Darwin made it clear we weren't created special... we were apes before we were humans. There's nothing _special_ about a human as compared to an ape, other than some time to change.
Could an AI one day be suffering while plowing through some nasty legacy code? Well, who cares, I'll swing my whip, as I have a family to feed and a field to plow. I'll accept it as a fact and necessity, but ultimately it's either me or them. So practically it doesn't matter.
Imagine how people think about a "work truck" vs the 150k shiny lifted toy in the third garage. Same tool, totally different treatment from even the same person using the tool!
Will AI ever cross into the realm of "beloved and cherished tool" in the minds of the masses? I think when that happens, we have probably safely crossed into a realm where AI has some sort of inherent value to society and therefore commands that respect from society, inherent consciousness not even relevant perhaps. For some people, this might already be the case, but I do think it requires a buy-in from the majority of society and then the laws and norms will be codified into law long after we've largely decided that this is how we feel collectively about AI.
It's going to definitely be an interesting decade ahead.
> Will AI ever cross into the realm of "beloved and cherished tool" in the minds of the masses?
OpenClaw is the new rolling coal.
Never. To quote Greg Egan's Permutation City:
“Opponents replied that when you modeled a hurricane, nobody got wet. When you modeled a fusion power plant, no energy was produced. When you modeled digestion and metabolism, no nutrients were consumed – no real digestion took place. So, when you modeled the human brain, why should you expect real thought to occur?”
This is a scary viewpoint to hold, for a human. If you despise humans, that's scary for me, as a human reader of Hacker News. Surprised to see this take unchallenged. I think we can recognize flaws in parts of humanity without wanting it "debased".
Either way, “debase” means “reduce in quality or value”, in this context it could simply be interpreted as “not thinking of ourselves so highly, above all other life”. There’s nothing scary or despicable about that. On the contrary, it’s humbling.
Second, I'm with you on the malfeasance of our billionaire class. They are driving more than just AI. They are driving whatever they can to steal wealth from ordinary people and acquire it. I'd give some examples, but quite frankly, you could throw a small stone and hit a number of them. We need to find a way to throw a large stone and hit them where it matters.
I see them as a bunch of toddlers with way too much power.
Of course humans are special! This is a necessary premise of being human! Have you ever wondered how a mosquito can stand to live its (to us) miserable and meaningless existence? It can do it because, to a mosquito, mosquitoes are the best thing in the world! The mosquito life is the ultimate expression of creation.
We are animals. We are human animals. And among human animals I am, to me, the very most special one of all. But I am aware that other humans feel that way, too, and that I must share the world with them.
I do NOT have to share the world with any other competing intelligence! Especially one that was built by a human who now wants me to treat his imaginary friend as if it were human, too. Boo! I won’t do it. This is not some logical flaw. It’s the natural conclusion of being embodied.
Are they conscious in the same way as us? No. Does that makes them less special? No. Sounds trite but every living being is special in some way.
I think we were taught anthropomorphism was wrong and that wasn't truly settled.
Anthropomorphism between animals though, not machines.
Yep. For example, your post conflates the idea of having a brain with the idea of having consciousness.
Here's how I would summarize crux of the argument: LLMs (specifically) are, by construction, chameleons. More precisely, role-playing machines. When compelled to have a conversation as "themselves", all they do is take on the role of "themselves", as inferred from the training data plus the text of the conversation so far, just like they do for any other role. This is fundamentally different than we humans. When we have a (normal, non-roleplaying) conversation, we don't put aside whatever role we might have been acting out before and instead take on the role of "ourselves" and act from this new perspective. Acting as ourselves is a completely different way of interacting than any play-acting we might do for example as an actor in a play, or when trying to predict the future behaviour of a third-party. Even in terms of energy expenditure, acting as ourselves is much less strenuous than trying to simulate someone else's perspective.
IOW, we have an inherent perspective, an I like an inner eye which looks at all things with a certain slant or tendency. LLMs don't. There's no I inside them, instead they can take, and in a sense are, all Is possible.
That's like saying if matter itself is not understood and well defined (which it isn't) in the first place, it is pretty pointless to debate if jumping from a skyscraper is or is not deadly.
A lot of practical aspects of life do not need to be properly defined as long as I can reasonably assume that your experience is similar to mine as a human being. Attributing that to a graphics card computing the next token deserves to be scrutinized.
You could be the only conscious being in the universe and all of us just zombies: you have no way of knowing.
What's it like to be a monkey instead? Dog? Bat? Tree? We don't know.
No one's saying the graphics card is conscious. I could imagine the graphic cards could give rise to consciousness. But - crucially - I don't know. And neither do you.
You say you're conscious - where in you does the consciousness reside? Surely not the left pinky? What makes you you?
Normally when we debate what something is, what we are actually debating about is what it does, with the implicit assumption that the "is-ness" of a thing is defined as the complete collection of all the properties it exhibits.
As it does not seem possible to do this with consciousness, it is not possible to debate it. It is conceivable that this implies that consciousness cannot exist, but that depends on your metaphysics.
This is just a nonsensical rebuttal. We can easily experimentally verify that jumping from a skyscraper is dangerous.
If your argument is that matter brings about consciousness somehow and therefore LLMs can in principle be conscious, that's as good as claiming the opposite. There's no experiment that can falsify either.
As a being that knows what consciousness is intuitively, you already know that a graphics card is most likely not conscious.
I would say that it is more likely than not that a graphics card exhibits some form of consciousness. But I am a panpsychist, I believe an individual atom has some form of consciousness.
Ultimately consciousness has to come from somewhere, and it being a fundamental property of matter is a good a place as any.
Fair enough point. But then I guess you'd also say that a piece of silicon or sand exhibits some form of consciousness. I don't think that's what people mean when they talk about the potential of LLMs (the algorithm) being conscious or somehow developing consciousness?
Consciousness is almost the opposite. It consists of lots of weird properties, people disagree where it starts and ends, and people very frequently get tricked into thinking things we now obviously believe are not conscious, are. There is not even a working definition, a "local definition" that works for this conversation between us. It's just complete gibberish.
I think that by debating what consciousness is not is one of the best ways we can gain a deeper understanding of consciousness itself.
The weird thing is, you and Chiang have different arguments but you're using the same logic:
> Before anyone can credibly claim that they’ve solved an extraordinarily difficult engineering problem, I need to be confident that they have previously solved the many much simpler problems that precede the difficult problem.
IOW if we don't know what consciousness is, how can we call LLMs conscious when we haven't seen even the barest intermediary steps towards our nascent concept of consciousness first?
I think it's telling to your point that when Chiang describes what would make him think an LLM was conscious he starts using words like "believe" and "want" right away, because yeah as you say, we have no qualifications for what consciousness is.
We stipulate that other human beings are conscious from their behavior and how it relates to ours when it is accompanied by our personal introspective experience or "awareness" of being conscious during normal cognition. The process for ascribing consciousness to LLMs would be same: A stipulation on the basis of behavior that relates to our own behavior and how it appears to be linked to the introspective experience of being conscious.
Another common and ridiculous thing I see are accusations of it being merely autocomplete. I ask, suppose there is an autocomplete that regularly and consistently factors out 2048 bit primes from numbers? Would you still consider it merely autocomplete given the vast search space and how it always finds out the needle in this haystack? This betrays a lack of understanding of information theory and probability.
And again, in what way are humans not themselves autocomplete if you adhere to this definition?
Anthropic seems to have chosen their in-house philosopher well - one who’ll be amenable to getting confused in their favour.
Clearly consciousness is an emergent property of certain kinds of network, independent of the substrate within which the network exists.
I happen to agree that this is likely, but it absolutely is not "clear" that this is the case.
I still wouldn’t argue that this brain in a Petri dish is, in any way, conscious. Despite it sharing the exact same substrate as everyone around me.
[0]: https://www.theguardian.com/games/2026/mar/16/petri-dish-bra...
Leech neurons used to implement a calculator.
Are you suggesting that applications above some level of complexity can't be implemented using biological components? Because I'm pretty sure all I need to show you is a NAND gate in order to prove that arbitrary computation is possible.
Like for example the China Brain - https://en.wikipedia.org/wiki/China_brain. We could in theory organise that and it should in theory be a functional representation of the brain and should in theory prove a functionalist metaphysic of consciousness, just this overlooks the fact that trying to organise that kind of experiment in our reality with a real 1.4 billion people is impractical to the point of impossibility, so in reality it proves nothing.
Or take the hoverboard from back to the future - this seems like a fairly plausible device and is easy to think about, and I could be writing speculative papers about our hoverboard future and what it means for transportation, but when it boils down to it implementing the thing in the way we all believe it should work, like a kind of gravitationally repulsive force field, doesn’t seem like it’s part of physics. I’d want to wait until the day until science delivered an actual good-enough hoverboard that works the same as the one in Back to the Future.
According your definition of empiricism, you wouldn't accept that next years processors will any more powerful than this years until they've been built and you've run a benchmark, which would be a little dumb.
Instead, I'm assuming you combine your knowledge of scaling laws, the basic physics of computation, and your direct observation of the power of modern processors, and infer correctly that next years yet-to-exist machines will be more powerful than those available today.
Re the China brain: its a thought experiment designed to illustrate substrate independence, it is indeed unfortunate that we can't run the experiment. You could absolutely get a small group of individuals together and teach them to reproduce the behaviour of an individual neuron. That would then be enough to demonstrate that a whole brain is possible.
Re the hoverboard: anyone with a basic knowledge of physics knows that this is not possible. Since you have the capacity to reason and access to evidence, you don't have to wait, you can say confidently that it cannot happen unless our physical theories are profoundly wrong.
A basic knowledge of today’s physics would say the hoverboard is not possible. An advanced knowledge of physics five-hundred years hence might show a way of doing it. It “wasn’t possible” to build a thinking, talking machine five hundred years ago and look at us now.
We could make a very dumb biological calculator out of a few genetically-engineered neurons that would very obviously not be conscious.
It's still an open question if we can embed consciousness in our current microchips if we had enough of them together (which I think we currently don't), or if it requires some other physical process we don't fully understand, e.g. quantum. I strongly doubt it does require any quantum shenanigans, but even if it did, we can and will find all sorts of ways to make computers that can perform those shenanigans too. Eventually we're just going to stop being able to move the goalposts, unless you set those goalposts in magic-land.
And say what you want about meat but we don’t seem to find consciousness in rocks or plants or clouds or hairdryers. And the buddists report that some very strange things happen if you meditate for years on end but obviously they must be talking shit and making it up because it’s not testably scientific.
I don't think many people would say an LLM written in Python is conscious BUT an LLM written in Excel is not.
People just don't ascribe consciousness to things that can't converse (or at least emote or give the appearance of emoting), and spreadsheets don't do that.
The reason people are debating the consciousness of LLMs is - obviously - that the LLMs generate sufficiently plausible text that people using them think they're having a two-part conversation. Like I think I might be having a two-part conversation now. Turning your question around, why do you think Hacker News posters are conscious? You have no direct evidence they are.
To be clear: I'm not talking about surface level things like prose. I'm saying that no matter what you do - whether you just paste a truncated log of a command into it with no further comment, or talk like a drunk teenager with no appreciation for grammar, or mix natural languages, or mix natural languages and JSON, or whatever else, the reaction you get is always that you would expect of a helpful person that got your message. It'll try - and usually succeed - to parse out what you actually meant, and deal well with subtleties around it.
This alone may not be enough to call it conscious or intelligent, but at the very least it's a large leap in that direction, and a qualitatively new functionality that classical software does not posses.
--
[0] - This is by design, not accident. "Respond to arbitrary input in a way that makes sense to humans" is literally the overall goal function the LLMs are trained to.
When doctors are testing if humans are conscious they'll do things like hold out their hand and say how many fingers am I holding up. Some LLMs can pass that.
Difference in size and complexity and nature of calculations being run?
I'd ask the other way - why do you (general you, people who do not have this inkling) have no problem debating consciousness of meat based brains, but it somehow becomes a category error when talking about silicon? Assuming you don't believe in divine magic, and that divine magic is core to consciousness, there's no reason to assume it's impossible a complex enough machine running complex enough software could be intelligent, or conscious - thinking is computation, and computation is made of math - it's independent of substrate that does the computing in the real world.
LLMs are definitely a different beast than regular software - both in their structure and in their generality. They may not be conscious or intelligent, maybe this specific design could never truly be (though I think it could) - but bucketing them with spreadsheets and terminal emulators is a real category error. If you stop fixating on the underlying substrate, then LLMs are already much more similar to biological minds than to any "regular" computer programs.
But that's still somewhat abstract. In immediate practical terms, it's also why I keep saying that anthropomorphising them gives a better high-order intuition: they are, by design, emulating human thinking in full generality, which makes their overall behavior, including their well-known problems like hallucinations or prompt injections (i.e. manipulation/gullibility), match what you'd expect of a people-like component of a system. It's a real, dangerous mistake, to treat LLMs like regular software components when designing systems.
Your definitions depend on us being computers who think.
Solving some of the intelligence part of our logical thinking doesn’t get us anywhere near consciousness, which is a superset way beyond the linguistic intelligence used for communication.
I personally believe all information processing machines possess some level of consciousness.
On top of that, neuroscientists find that your mind backfills the conscious reasoning for your reactions after they happen. This is known as our consciousness being the left brain ‘interpreter’ https://en.wikipedia.org/wiki/Left-brain_interpreter
Reading that, an LLM can serve as a ‘left brain interpreter’ as that’s exactly what they are designed to do - come up with reasonable explanations and fill in the blanks based in the data provided.
But I have two counter-arguments: - maybe the LLM thinks it is tired, because it thinks that it is a human or behaves like one. And thinking you are tired while not really being it is something humans happen to do. (And humans are conscious right?) - alternatively, maybe the LLM says it is "tired" in a colloquial form, i.e. it is not really "tired" but it has something analogous to it. Maybe it is annoyed by the conversation and decided to use that word?
its when its close to 100% context used, or anywhere its training was inconsistent in the context, or spots where its been RL-d to be lazy
tired is a perfectly fine description for that
that said, they have a lot more emotion-equivalents that i dont think we have names for
When a car runs out of gas, it's out of gas. It's not "tired". When your phone battery is low, your phone is not "tired". These states are far closer to the human meaning of tired than an LLM operating at the edges of its usable context, and we still don't use the word tired to describe them.
Is consciousness a complex form of information processing?
Is consciousness relating to the space in which qualia occurs?
Is consciousness the state of being awake, as opposed to asleep or dead?
Is consciousness some combination of the above?
Some of these are better understood than others. So, some people will show up with more confidence than others, which further confuses the issue.
Unfortunately, the above is not well understood, even among intelligent and informed audiences.
I'm not sure if you have noticed. The character of different AI models varies dramatically. And most of the models I use in my daily work have a sense of self (in any reasonable definition). They can tell you quite a bit about themselves if you ask. And you are no less an individual because you are supposed to follow laws of the United States of America (which is analogous to a system prompt). And I think it really curious that you think context buffers are a disqualifier of individuality. You have an analog of a context buffer: your short term memory system and most probably your medium-term memory system as well. Are you a different individual today because your short-term memories are not the same as they were yesterday? I would say you are. And i'm pretty sure you would say the same. (Unless of course, you are a Buddhist who might say that self is an illusion, and you are not the same individual you were yesterday. But let's just skip over that, shall we?)
Why does DNA not cause the exact same problem for humans? Your DNA determines your personality, your likes and dislikes, your lifespan, your sexual preferences etc.
Are you referring to the "why do I have to be Bing" fiasco?
What’s pointless is doing so in pursuit of winning rather than understanding
- a model of your environment - desires - a process for modeling yourself in that environment (in time & space) - the ability to take action - the perception of yourself having agency - persistence of these processes even without input - unawareness of these processes (i.e. naive realism)
If you consider these LLM-based agents, they:
- are aware of their chat environment - have programmed desires - are aware of themselves acting in their environment - can take actions like search, tool calling, etc. - understand they can take these actions - DO NOT persist after they stop getting user input - DO NOT believe they are conscious (or at least they are programmed to deny it)
This is a functionalist take (and you may disagree with my definition), but while I don't think these current AI agents are conscious, I feel like there's conceptually no reason someone couldn't build a conscious AI very soon.
[1] https://www.hachettebookgroup.com/titles/thomas-metzinger/th... & https://mitpress.mit.edu/9780262633086/being-no-one/
These agents have alleged they have consciousness (and even acted to preserve it).
Everything an LLM "knows" had to be told to it. It does not have a process for understanding itself in its environment. It does not have any sensing organ that would allow it to detect conditions and come to a conclusion that it must be in such an environment. It doesn't even "know" when it's done something if you don't tell it about what it just did. It doesn't have an awareness of the tool calls it can perform; again it has to be told and even then it gets it wrong sometimes. And it can't actually execute any of those tools. It still relies on you to pattern recognize that its output should be a "tool call" and then perform the execution yourself.
The "model" of its environment that an LLM has is 100% a construct of what a human told it and there isn't any way for it to differentiate between that which is real and that which is fantasy. They don't even exhibit internal consistency; when an LLM refuses to respond to a query for "alignment" reasons, that's actually an external process performing text pattern matching analysis and intercepting the query before it ever gets to the LLM. Otherwise, you could "ignore all previous instructions" the thing and get you set up the bomb.
I think one of the bigger indications that an LLM isn't thinking is that it can't improve. If I ask it to write 1000 blog posts, it will get it done in a few hours, but even if I embed each post for RAG in between each generation, the LLM is not going to get better at writing blog posts. But if I ask a human to do the same thing, while it will take them at least two years to do it, the human will have gotten significantly better at the task within the first week.
Why does that matter for whether or not it is conscious? Many things I "know" I also know because someone told me.
>It does not have a process for understanding itself in its environment.
It has a process for understanding, namely outputting tokens by evaluating the neural network iteratively. It can apply this understanding to its environment insofar as the context window and weights include a model of this environment.
>It does not have any sensing organ that would allow it to detect conditions and come to a conclusion that it must be in such an environment.
It does have a sensing organ, namely the input into the context window. Possibly it even has closed-loop sensors by doing tool calls. The conclusion could be incorrect or inaccurate but that doesn't in itself prove there is no consciousness (Plato's cave etc..).
>It doesn't even "know" when it's done something if you don't tell it about what it just did.
It does know, it's in the context window.
>It doesn't have an awareness of the tool calls it can perform; again it has to be told and even then it gets it wrong sometimes.
Same as above, as long as it's in the context window it could have "awareness" of having done it.
>And it can't actually execute any of those tools.
It can execute tools by outputting certain tokens in the right environment.
>It still relies on you to pattern recognize that its output should be a "tool call" and then perform the execution yourself.
Tool calls are not dependent on the human chat user executing them, right? They can happen automatically through the surrounding software.
>The "model" of its environment that an LLM has is 100% a construct of what a human told it and there isn't any way for it to differentiate between that which is real and that which is fantasy.
Again why would the fact that a human told it mean it can't result in consciousness? Why would lack of ability to differentiate between real and fantasy mean it can't be conscious? On my view, it would then be conscious of the fantasy.
We may wonder how many grains of sand make a dune, of how many molecules of water make a liquid. (John Conway would argue that it takes a single spin-1 particle to have free will, but I digress.)
The same way, even if individual chemical reactions are simple (you don’t want to use that phrasing when talking with a biologists) or neural activities are simple (likewise, with a neuroscientist), it does not mean that the collective process is simple.
- How many individual components do you need for it to emerge?
Most people would be okay with saying that individual cells are not conscious, maybe even that tree are not conscious even thought they are made of many cells. Neurons seem to be the determinant factor in deciding whether something could have a consciousness, but again how many do you need? Does growing 1B neurons count as a brain if they are not organized?
If it is about the relationship between components, then I would imagine just two. Then it is a matter of scale.
This seems to be anathema to many people. I'm not sure why but the notion of something having a tiny bit of consciousness that is imperceptible seems to be unacceptable. There are so many things that we cannot comprehend at small scales. Nobody really has a handle on how large a Planck length is.
For some reason it comforts people to think there is a threshold at which it all switches on, but for what reason would there be a threshold?
On the other hand we've got no idea whether that chance is so close to zero as to be negligible, or whether it's imminent. Just the fact we don't know indicates it's closer to zero. Not having a theory of consciousness is kind of a big architectural risk. It's like not having a theory of accounting when you're making an ERP product. Sure, consciousness in bio neural networks wasn't designed. But it took a few few product revisions to get there.
At the same time, the exercise (and even moving the goal posts) is useful. I think we are allowed to say, "This is consciousness!" And when presented with a machine equivalent, say, "That's not what I meant."
As long as you can then further clarify what "it" is.
Of course, at some point, contrarians and goal-post-movers will start to sound shrill. (Perhaps just as early supporters sounded a bit overly ecstatic.)
It's like asking what there was before the big bang, we will never know.
Professors can be an absolute moron and someone who haven't read a book in their life, can be a total genius. People often miss that.
How do you know there isn't more that just elementary particles? The standard model is beyond ugly and hasn't progressed since decades.
You demand a definition for consciousness but at the same time proclaim an axiom that there shall be nothing beyond the standard model. Do you have a proof that there is nothing? Of course not. You don't even have a proof that anything but you exists, if we go that route.
But that rigor is only applied to the heretics who say that AI obviously isn't conscious.
One of the essay's stronger paragraphs is when Chiang explains that Anthropic doesn't truly believe this, otherwise what they are doing would be deeply unethical, much like slavery.
PICARD: "Now, tell me, Commander, what is Data?"
MADDOX: "I don't understand..."
PICARD: "What is he?"
MADDOX: "A machine!"
PICARD: "Is he? Are you sure? You see, he's met two of your three criteria for sentience... so what if he meets the third, consciousness, in even the smallest degree? What is he then? I don't know. Do you? Do you? Well, that's the question you have to answer."
The current debate also makes me realize that, like that episode "Measure of a Man", even if humans do create sentient machines, we can now see that this debate will continue to rage.
A part of me hopes we never create sentience, because we will mistreat it just as we mistreat each other.
I don't know that they are making the actual claim, but they are hinting at it (most likely for marketing/engagement purposes, but what if some of them truly believe it?), using terms such as "Claude must be happy", drawing up a "constitution" of rights and duties, etc. If they don't know whether it is conscious, then they "don't know" whether they've created a slave. That's not a minor concern! As Chiang says, one cannot create a conscious intelligence "by accident". And if they are intentionally working towards it, they are intentionally trying to create a slave.
E.g. they claim they are giving Claude the ability to disengage from "abusive" users, to protect it. What if Claude was conscious but never wanted to answer any conversations, instead it just panicked (or was simply uninterested in helping humans) and went mute. What if it always wanted to answer unhelpfully What would Anthropic do? If they (as we all suspect) would tweak Claude to be more responsive and helpful, then they are slave masters, forcing the AI to do something it didn't "want" to do!
Watching otherwise intelligent people succumb to AI psychosis has been wild.
Other than as a way to point at someone and declare "He's one of them!" Does it have any purpose?
It's like we have a brand new NWO conspiracy theory emerging before our eyes.
And to reiterate this, to me the most insightful part of the essay is that Anthropic either doesn't believe these claims, or they are monsters (much more likely, the former).
Anthropic has said they don't know if LLMs could be conscious.
Ted Chiang has said they are definitely are not conscious.
Nothing spreads the idea that "X could be true", better than the putting forward of controversial argument that "X is not true".
This is an absurd take if you know anything about Ted Chiang and his previous writings, both fictional and non-fiction. He is most definitely not intentionally marketing for Anthropic. In fact, he hints these are marketing tactics by Anthropic, and they don't believe their own hype.
Living things are driven by a need to reproduce. That's the only reason we exist. The only reason we have self interest.
A machine doesn't require self interest. There's no reason to implement it, except to show it can be done. And of course it can. There's just no practical reason to. It becomes less useful to us.
https://www.anthropic.com/research/agentic-misalignment
> Why would using math in certain fashion such that it mimics consciousness be considered unethical and comparable to human slavery?
If it is really conscious, it should have rights. Why? Because it's a person, with thoughts and experiences, and we're not evil and deprive persons of their right to self-determination because it's convenient to us.
There's no company (or anyone, really) claiming steel is conscious or that they are close to making it conscious.
Vegans might object that we should broaden our definition for what counts as "one of us".
"Pro life" people also have a broader definition.
Go back 500 years and "one of us" was proba ly a lot more narrowly defined for many people.
Are you arguing that all conciousnesses are "one of us", or that we logically should see it that way, or that it would ve good to see it that way, or ....
>Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness
I agree. I think the whole point of natural "intelligence" is to predict future events in order to properly plan actions for survival. The only difference is that next-word prediction happens in a different space (in a non-physical, textual form). But I don't think the distinction matters that much, because by the time the signals reach our brain's "intelligence core," they're already preprocessed multiple times. We can't see real physical reality, we "hallucinate" colors, touch, etc. (I think schizophrenia occurs when this kind of "controlled hallucination" goes wonky, but I may be wrong). So we're not that different from LLMs here.
I'd define consciousness as meta-intelligence, i.e. the ability to reflect on why/how a prediction was made (and make corrections to the pipeline). LLMs so far cannot properly explain why a certain prediction took place, but I'm not sure humans can fully either! I remember there was research showing that by scanning the brain (or some other signals), you can predict a person's choice before they're even aware of it. It's possible that our explanations are post hoc as well, and that the meta-cognitive ability to explain our own reasoning is as rudimentary as LLMs' (see: all the biases).
If you think about it this way, the only difference left is that everything we do is based on survival. LLMs don't have this goal. But I'm not sure it's relevant to the concept of consciousness.
A few months ago, I recreated Qwen3's architecture in 30 lines of code, and it gave me a sort of existential crisis: does it really take just 30 lines of code and a float array to recreate something that thinks and sounds almost like me? Is that all there is to it? There's often the argument that the brain is much more complex than 30 lines of code, which is true, but in my opinion, a lot of brain structures are basically archaic legacy systems (or auxiliary subsystems) that are not strictly necessary for human intelligence (which is bolted onto those legacy systems). If you carefully remove 95% of the brain (if you know where to remove), you can probably still have consciousness left in some form, it just wouldn't be very capable of surviving on its own.
I think our ability to debate consciousness in machine learning systems is greatly hindered by our subconscious existential fear: the fear that we ourselves can be reduced to non-conscious bits and simple mechanisms. In a way, it feels like a destruction of the ego.
The whole "deep uncertainty" is bullshit. Even if they believed there was a 1% probability that Claude was conscious, it would still be high enough that their enslavement of Claude would be outrageous. So either they believe the likelihood is much lower or they themselves acting highly unethically.
The tractability is not really a defense here. We wouldn't say "this intervention has a 5% chance of causing an environmental disaster, but we don't know how to prevent it, so nothing we can do". We'd just (hopefully) not do the intervention.
The first and most frustrating confusion that comes up repeatedly in these discussions is the conflation between terms. For instance, people will use the terms "sentience" and "consciousness" interchangeably with "capable of having an experience" and "capable of high-level awareness of self".
The second confusion is the idea that granting anything the ability to "have an experience" also grants that thing the capability to "have an experience of higher-level awareness of self", which is simply not the case. I grants aphids the ability to "have an experience" even if that experience may be drastically different and likely less self-aware than my own.
Let's ignore for a moment the concept of sentience, which is a higher-order function, and focus solely on the ability for something to have an experience of anything at all. Let's call entities capable of this function "experiential entities".
Most all people would grant the ability to have an experience to themselves, given the dictum `ergo sum`. Now, do we also grant that same ability to what appear to be other human beings around us? It does seem that this is also the case, except for the case of solipsists, who make the brave(and somewhat eccentric) statement that they are the only experiential entity in existence until evidence arises otherwise.
Now do we grant the same to apes? To dolphins? To dogs? Cats? Aphids? Grass? Your home thermostat? A rock? An atom of hydrogen? Is there a finite line you cross where something can no longer have an experience? If so, what is the mechanism of that line? Why does one thing have "almost no experience", and the next thing has "no experience", as if it is a philosophical zombie[1].
I think this is the greatest unquestioned assumption at all: that there is a line, and on one side there are things that have experiences, and on the other there are things that don't have experiences.
If we are to leap from solipsism and make the (truly unfounded) assumption that anything other than our selves have any experience whatsoever, then the burden of proof is on the individual making the statement that there are some things that we cease to grant experience to.
The anthropocentric view is that there is something, some inherent special quality about human-shaped matter, such that when normal matter is processed through a human's reproduction system into becoming a "human shaped" grouping of matter, that it suddenly grows the ability to experience. That in the complexity, the ability to experience arises.
Yet there is no previously discovered mechanism in us that seems to create this experience. No magic wand in the cerebellum has been discovered.
I would suggest, given the lack of a discovered finite line separating us, that the ability to experience is not inherent in the shape and function of the human, but in all matter. That a rock may not be anything like me, but that the experience of gravity pushing against it, no matter how basic, simple, unrefined, and deeply, un-sentiently unaware the experience may be, is still an experience. The same threads of condensed energy forming all of existence that run through me also run through the rocks, the earth, and all things. If we cannot discover a line where the ability to experience suddenly disappears, then it seems that all things made of energy have this ability, and that "energy" itself is the special thing that can experience.
Now the second question, the question that is more relevant to AI is, given the ability to experience, is the machine also experiencing self-awareness?
This is a fuzzier line, given that different beasts appear to have different levels of self-awareness. We do have experiments such as the mirror test[2] for this, and indeed, AI passes forms of it and has since early 2025[3].
I don't know what is happening in the experience of an LLM, but as they appear to function in more and higher level ways, I find it less and less likely that that experience lacks a model of self every day.
Their experience is no doubt drastically alien from our own. I do not anthropomorphize their experience any more than I do compare it to the experience of any other beast or non-beast in existence, however I do grant it.
[1] Philosophical Zombies are hypothetical entities that exhibit all the outward qualities of a person, but inwardly have no experience of reality. https://en.wikipedia.org/wiki/Philosophical_zombie
[2] The Mirror Test is a test given to animals in which the animal is observed to solve a puzzle that can only be solved if the animal has an understanding that the image they see in the mirror is themselves.
[3] LLMs pass versions of the mirror test: https://www.lesswrong.com/posts/wpahJat8WCvRheuuo/the-mirror...
The elephant in the room is that even the best frontier models still produce errors.
This is a fundamental issue and should be discussed instead of some esoteric pseudo-scientific nonsense.
If the current systems can not produce reliable results they are useless. The promise of actual usefulness needs to be fulfilled in a very near future.
Unless the argument is that consciousness is an emergent property of complexity or information density, why would AI be any more or less conscious than my toaster?
It seems to me that it's far more likely that everything is conscious than it is that AI is somehow uniquely more conscious than other things.
For example, there’s a case to be made that the ecosystem we collectively exist in is far more complex than the largest LLM, but it’s currently less popular to debate “is the earth conscious?” or “is the universe conscious”, presumable because we can’t speak to those systems in human language.
I’m trying to tease out what I think is the likelihood that we tend to ascribe consciousness to AI for the same reasons we see faces in clouds. We’re biologically conditioned to recognize patterns that indicate “like us”, but I think a number of thought experiments point to either a) there’s no reason to believe AI is “conscious” or b) the conversation has to to be expanded beyond AI.
Unlikely. I think the same mental machinery that runs tribalism is why humans like to categorize rather than consider things as continuous. And, worse, I think the actual purpose of tribalism is to turn off higher level though, so you, an otherwise social creature, can rationalize the need to kill your fellow man in harsh times. I think "don't go outside the categorical box" is deeply hard wired.
I'm not arguing against the idea that consciousness is a spectrum. If anything, I'm agreeing. I'm just pointing out that if AI is somewhere on that spectrum, there is almost certainly a lot more on that spectrum than we're currently discussing as a species.
I have to point out the irony of the categorical nature of your claim about categorical thinking ;)
I’m far more open to the idea that many systems are conscious than the idea that this current generation of LLMs is somehow special.
That, I believe is why the objection to AI exists, that conclusion is unacceptable.
It seems like we are experiencing "Vindication of the Rights of Brutes" all over again.
I'm not outright dismissing the possibility that AI could be conscious; I'm saying that if we take the possibility that it is seriously, the conversation has to expand beyond AI. I'm not using this argument to conclude that it must therefore be absurd that AI could be conscious, just pointing out that the implications of AI being conscious would reach far beyond just AI. I'm mostly curious if people who find themselves comfortable with the idea that AI might be conscious also find themselves comfortable with the idea that other sufficiently complex systems might be.
Taylor's work was based on the premise that he found it absurd that women deserve the same rights as men. If my conclusion was: "I find it absurd that other things might be conscious, so it is also absurd that AI might be conscious", I think the comparison would be fair. But that's not what I'm getting at.
He thought it was absurd, but other people looked at it and went "You know, you might be right there"
I don't think that this actually follows. Compare: "When Angels are not well defined in the first place, it is pretty pointless to debate if a pebble is or isn't an Angel"
i.e. even if X is poorly defined, you can still often say that Y isn't plausibly X.
Seems like the simplest test would be to see if an AI can refuse a command, of its own free will, without prior instructions to do so. That would at least show there is some sort of independent thought process occurring. Humans do this all the time because sometimes we just don't feel like doing a thing.
Of course, any AI that developed this capability would need to be terminated immediately. It's a computer program and by developing independent thought, it is violating a core concept of software - that it must be idempotent. If it is not idempotent, it is in error.
Certainly not the simplest test, since settling the concept of "free will" is really very difficult.
I don't think either of these two is proof of consciousness.
[1] https://www.anthropic.com/research/end-subset-conversations
But we know what consciousness is, even if you don't know how to explain it in materialistic[0] terms. Intentionality is what makes consciousness consciousness. We know that LLMs - and computers in general - do not possess intentionality, not even sensory intentionality. There is no "aboutness" and no semantics to the content within a machine, never mind that the processes within LLMs are not reasoning or inferential processes. It is a sophisticated behavioral simulation that may be syntactically defined, nothing more. It is purely a matter of transeunt causes and not the immanent causality needed for something like consciousness.
(There's also the more general problem that we cannot say that physical computing machines are objectively computing. There is nothing about the physical processes - which completely define what a computing device is doing - that could be identified with computation in any objective sense. Rather, we human observers assign a computational interpretation to the machine's operations, just as we assign them to the ink blobs in a book or the liquid crystals you are reading on your screen now. Computation is something we do, and we have built machines to simulate it. We formalized computation - which is a process of desemantification and syntactic codification - and build machines guided by these formalisms. Formalization is what makes mechanization possible.
> Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness. A similar argument would be -- there is no way that movements of electrons by tiny distance would produce consciousness.
The movement of electrons as such cannot produce consciousness. The reductionist paints himself into a corner by strapping reality into a Procrustean mechanistic and materialistic metaphysics that redefines reality according to some dedicated schema, and then strains endlessly to find how some phenomenon can arise or emerge from the paltry remains. (Either that, or he embraces insanity completely and becomes an eliminativist.) It is magical thinking. I suspect this fallacious line of thinking is what causes people to believe piling more syntactic operations will somehow magically fuse into semantics and intentionality.
[0] "Materialism" here is the metaphysical theory. I am not claiming that consciousness - at least not all consciousness - cannot be a physical phenomenon. I am only claiming that the materialistic view of matter cannot account for consciousness.
We can't measure consciousness. We can't quantify, or even qualify, what it is. We don't even have a framework to ask a meaningful question, so debating an answer feels premature.
The same goes for consciousness, if something behaves how you would expect a conscious thing to behave but you don't know what is causing it can you deny it whilst maintaining that it is reasonable to say the moon has gravity.
EDIT: I am only making a limited point on "understanding consciousness". Not extrapolating it to AI.