I also think different ideas get conflated. It may be possible to build a machine that is super-human in the sense it can outperform the human brain in all kinds of measurable ways. Does not imply it possesses all the same qualities of the brain.

I respect a number of things Anthropic has published about the ethical issues at stake. But, having an in-house philosopher does invite you to make all kinds of unfalsifiable claims.

I think that by debating what consciousness is not is one of the best ways we can gain a deeper understanding of consciousness itself.

The weird thing is, you and Chiang have different arguments but you're using the same logic:

> Before anyone can credibly claim that they’ve solved an extraordinarily difficult engineering problem, I need to be confident that they have previously solved the many much simpler problems that precede the difficult problem.

IOW if we don't know what consciousness is, how can we call LLMs conscious when we haven't seen even the barest intermediary steps towards our nascent concept of consciousness first?

I think it's telling to your point that when Chiang describes what would make him think an LLM was conscious he starts using words like "believe" and "want" right away, because yeah as you say, we have no qualifications for what consciousness is.

Another common and ridiculous thing I see are accusations of it being merely autocomplete. I ask, suppose there is an autocomplete that regularly and consistently factors out 2048 bit primes from numbers? Would you still consider it merely autocomplete given the vast search space and how it always finds out the needle in this haystack? This betrays a lack of understanding of information theory and probability.

And again, in what way are humans not themselves autocomplete if you adhere to this definition?

That's like saying if matter itself is not understood and well defined (which it isn't) in the first place, it is pretty pointless to debate if jumping from a skyscraper is or is not deadly.

A lot of practical aspects of life do not need to be properly defined as long as I can reasonably assume that your experience is similar to mine as a human being. Attributing that to a graphics card computing the next token deserves to be scrutinized.

Anthropic seems to have chosen their in-house philosopher well - one who’ll be amenable to getting confused in their favour.

Is consciousness a complex form of information processing?

Is consciousness relating to the space in which qualia occurs?

Is consciousness the state of being awake, as opposed to asleep or dead?

Is consciousness some combination of the above?

Some of these are better understood than others. So, some people will show up with more confidence than others, which further confuses the issue.

Unfortunately, the above is not well understood, even among intelligent and informed audiences.

- a model of your environment - desires - a process for modeling yourself in that environment (in time & space) - the ability to take action - the perception of yourself having agency - persistence of these processes even without input - unawareness of these processes (i.e. naive realism)

If you consider these LLM-based agents, they:

- are aware of their chat environment - have programmed desires - are aware of themselves acting in their environment - can take actions like search, tool calling, etc. - understand they can take these actions - DO NOT persist after they stop getting user input - DO NOT believe they are conscious (or at least they are programmed to deny it)

This is a functionalist take (and you may disagree with my definition), but while I don't think these current AI agents are conscious, I feel like there's conceptually no reason someone couldn't build a conscious AI very soon.

[1] https://www.hachettebookgroup.com/titles/thomas-metzinger/th... & https://mitpress.mit.edu/9780262633086/being-no-one/

On the other hand we've got no idea whether that chance is so close to zero as to be negligible, or whether it's imminent. Just the fact we don't know indicates it's closer to zero. Not having a theory of consciousness is kind of a big architectural risk. It's like not having a theory of accounting when you're making an ERP product. Sure, consciousness in bio neural networks wasn't designed. But it took a few few product revisions to get there.

What’s pointless is doing so in pursuit of winning rather than understanding

At the same time, the exercise (and even moving the goal posts) is useful. I think we are allowed to say, "This is consciousness!" And when presented with a machine equivalent, say, "That's not what I meant."

As long as you can then further clarify what "it" is.

Of course, at some point, contrarians and goal-post-movers will start to sound shrill. (Perhaps just as early supporters sounded a bit overly ecstatic.)

We may wonder how many grains of sand make a dune, of how many molecules of water make a liquid. (John Conway would argue that it takes a single spin-1 particle to have free will, but I digress.)

The same way, even if individual chemical reactions are simple (you don’t want to use that phrasing when talking with a biologists) or neural activities are simple (likewise, with a neuroscientist), it does not mean that the collective process is simple.

The whole "deep uncertainty" is bullshit. Even if they believed there was a 1% probability that Claude was conscious, it would still be high enough that their enslavement of Claude would be outrageous. So either they believe the likelihood is much lower or they themselves acting highly unethically.

The tractability is not really a defense here. We wouldn't say "this intervention has a 5% chance of causing an environmental disaster, but we don't know how to prevent it, so nothing we can do". We'd just (hopefully) not do the intervention.

Professors can be an absolute moron and someone who haven't read a book in their life, can be a total genius. People often miss that.

The first and most frustrating confusion that comes up repeatedly in these discussions is the conflation between terms. For instance, people will use the terms "sentience" and "consciousness" interchangeably with "capable of having an experience" and "capable of high-level awareness of self".

The second confusion is the idea that granting anything the ability to "have an experience" also grants that thing the capability to "have an experience of higher-level awareness of self", which is simply not the case. I grants aphids the ability to "have an experience" even if that experience may be drastically different and likely less self-aware than my own.

Let's ignore for a moment the concept of sentience, which is a higher-order function, and focus solely on the ability for something to have an experience of anything at all. Let's call entities capable of this function "experiential entities".

Most all people would grant the ability to have an experience to themselves, given the dictum `ergo sum`. Now, do we also grant that same ability to what appear to be other human beings around us? It does seem that this is also the case, except for the case of solipsists, who make the brave(and somewhat eccentric) statement that they are the only experiential entity in existence until evidence arises otherwise.

Now do we grant the same to apes? To dolphins? To dogs? Cats? Aphids? Grass? Your home thermostat? A rock? An atom of hydrogen? Is there a finite line you cross where something can no longer have an experience? If so, what is the mechanism of that line? Why does one thing have "almost no experience", and the next thing has "no experience", as if it is a philosophical zombie[1].

I think this is the greatest unquestioned assumption at all: that there is a line, and on one side there are things that have experiences, and on the other there are things that don't have experiences.

If we are to leap from solipsism and make the (truly unfounded) assumption that anything other than our selves have any experience whatsoever, then the burden of proof is on the individual making the statement that there are some things that we cease to grant experience to.

The anthropocentric view is that there is something, some inherent special quality about human-shaped matter, such that when normal matter is processed through a human's reproduction system into becoming a "human shaped" grouping of matter, that it suddenly grows the ability to experience. That in the complexity, the ability to experience arises.

Yet there is no previously discovered mechanism in us that seems to create this experience. No magic wand in the cerebellum has been discovered.

I would suggest, given the lack of a discovered finite line separating us, that the ability to experience is not inherent in the shape and function of the human, but in all matter. That a rock may not be anything like me, but that the experience of gravity pushing against it, no matter how basic, simple, unrefined, and deeply, un-sentiently unaware the experience may be, is still an experience. The same threads of condensed energy forming all of existence that run through me also run through the rocks, the earth, and all things. If we cannot discover a line where the ability to experience suddenly disappears, then it seems that all things made of energy have this ability, and that "energy" itself is the special thing that can experience.

Now the second question, the question that is more relevant to AI is, given the ability to experience, is the machine also experiencing self-awareness?

This is a fuzzier line, given that different beasts appear to have different levels of self-awareness. We do have experiments such as the mirror test[2] for this, and indeed, AI passes forms of it and has since early 2025[3].

I don't know what is happening in the experience of an LLM, but as they appear to function in more and higher level ways, I find it less and less likely that that experience lacks a model of self every day.

Their experience is no doubt drastically alien from our own. I do not anthropomorphize their experience any more than I do compare it to the experience of any other beast or non-beast in existence, however I do grant it.

[1] Philosophical Zombies are hypothetical entities that exhibit all the outward qualities of a person, but inwardly have no experience of reality. https://en.wikipedia.org/wiki/Philosophical_zombie

[2] The Mirror Test is a test given to animals in which the animal is observed to solve a puzzle that can only be solved if the animal has an understanding that the image they see in the mirror is themselves.

[3] LLMs pass versions of the mirror test: https://www.lesswrong.com/posts/wpahJat8WCvRheuuo/the-mirror...

One of the essay's stronger paragraphs is when Chiang explains that Anthropic doesn't truly believe this, otherwise what they are doing would be deeply unethical, much like slavery.

>Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness

I agree. I think the whole point of natural "intelligence" is to predict future events in order to properly plan actions for survival. The only difference is that next-word prediction happens in a different space (in a non-physical, textual form). But I don't think the distinction matters that much, because by the time the signals reach our brain's "intelligence core," they're already preprocessed multiple times. We can't see real physical reality, we "hallucinate" colors, touch, etc. (I think schizophrenia occurs when this kind of "controlled hallucination" goes wonky, but I may be wrong). So we're not that different from LLMs here.

I'd define consciousness as meta-intelligence, i.e. the ability to reflect on why/how a prediction was made (and make corrections to the pipeline). LLMs so far cannot properly explain why a certain prediction took place, but I'm not sure humans can fully either! I remember there was research showing that by scanning the brain (or some other signals), you can predict a person's choice before they're even aware of it. It's possible that our explanations are post hoc as well, and that the meta-cognitive ability to explain our own reasoning is as rudimentary as LLMs' (see: all the biases).

If you think about it this way, the only difference left is that everything we do is based on survival. LLMs don't have this goal. But I'm not sure it's relevant to the concept of consciousness.

A few months ago, I recreated Qwen3's architecture in 30 lines of code, and it gave me a sort of existential crisis: does it really take just 30 lines of code and a float array to recreate something that thinks and sounds almost like me? Is that all there is to it? There's often the argument that the brain is much more complex than 30 lines of code, which is true, but in my opinion, a lot of brain structures are basically archaic legacy systems (or auxiliary subsystems) that are not strictly necessary for human intelligence (which is bolted onto those legacy systems). If you carefully remove 95% of the brain (if you know where to remove), you can probably still have consciousness left in some form, it just wouldn't be very capable of surviving on its own.

I think our ability to debate consciousness in machine learning systems is greatly hindered by our subconscious existential fear: the fear that we ourselves can be reduced to non-conscious bits and simple mechanisms. In a way, it feels like a destruction of the ego.

The elephant in the room is that even the best frontier models still produce errors.

This is a fundamental issue and should be discussed instead of some esoteric pseudo-scientific nonsense.

If the current systems can not produce reliable results they are useless. The promise of actual usefulness needs to be fulfilled in a very near future.

Unless the argument is that consciousness is an emergent property of complexity or information density, why would AI be any more or less conscious than my toaster?

It seems to me that it's far more likely that everything is conscious than it is that AI is somehow uniquely more conscious than other things.

I don't think that this actually follows. Compare: "When Angels are not well defined in the first place, it is pretty pointless to debate if a pebble is or isn't an Angel"

i.e. even if X is poorly defined, you can still often say that Y isn't plausibly X.

But we know what consciousness is, even if you don't know how to explain it in materialistic[0] terms. Intentionality is what makes consciousness consciousness. We know that LLMs - and computers in general - do not possess intentionality, not even sensory intentionality. There is no "aboutness" and no semantics to the content within a machine, never mind that the processes within LLMs are not reasoning or inferential processes. It is a sophisticated behavioral simulation that may be syntactically defined, nothing more. It is purely a matter of transeunt causes and not the immanent causality needed for something like consciousness.

(There's also the more general problem that we cannot say that physical computing machines are objectively computing. There is nothing about the physical processes - which completely define what a computing device is doing - that could be identified with computation in any objective sense. Rather, we human observers assign a computational interpretation to the machine's operations, just as we assign them to the ink blobs in a book or the liquid crystals you are reading on your screen now. Computation is something we do, and we have built machines to simulate it. We formalized computation - which is a process of desemantification and syntactic codification - and build machines guided by these formalisms. Formalization is what makes mechanization possible.

> Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness. A similar argument would be -- there is no way that movements of electrons by tiny distance would produce consciousness.

The movement of electrons as such cannot produce consciousness. The reductionist paints himself into a corner by strapping reality into a Procrustean mechanistic and materialistic metaphysics that redefines reality according to some dedicated schema, and then strains endlessly to find how some phenomenon can arise or emerge from the paltry remains. (Either that, or he embraces insanity completely and becomes an eliminativist.) It is magical thinking. I suspect this fallacious line of thinking is what causes people to believe piling more syntactic operations will somehow magically fuse into semantics and intentionality.

[0] "Materialism" here is the metaphysical theory. I am not claiming that consciousness - at least not all consciousness - cannot be a physical phenomenon. I am only claiming that the materialistic view of matter cannot account for consciousness.

Seems like the simplest test would be to see if an AI can refuse a command, of its own free will, without prior instructions to do so. That would at least show there is some sort of independent thought process occurring. Humans do this all the time because sometimes we just don't feel like doing a thing.

Of course, any AI that developed this capability would need to be terminated immediately. It's a computer program and by developing independent thought, it is violating a core concept of software - that it must be idempotent. If it is not idempotent, it is in error.

EDIT: I am only making a limited point on "understanding consciousness". Not extrapolating it to AI.

LLMs process language. I'd even go so far as to say that LLMs "think" and "understand", or at least, they produce a facsimile of thinking and understanding such that it's useful for us to reason about LLMs as if they think and understand. We're not used to interacting with a non-human entity with the capability to process language, so it's easy to ascribe human traits to these things. But their "minds" (insofar as they have anything like a mind) are completely different from ours. These things have language without consciousness.

Chimpanzees are conscious. Dogs are conscious. Maybe ravens and cephalopods? Who knows. These animals do have minds much like ours. Higher order animals are conscious even if they don't have language.

If I'm going to be honest most of the people who advocate this type of thing tend to be, shall we say, crypto-duelists who really believe in a soul but not like intellectually but intuitively and keep trying to come up with excuse with it's not just meat. So like you can find philosophers advocating stuff like this but they tend to have a bit of an agenda.

You can't use AI atm without a human with proper knowledge spot checking and directing it. We should market it that way.

But honestly I doubt that there is a real requirement for that, surely you could just increase the resolution you're running the simulation at until the difference decreases sufficiently. Imagine using 256 bit floats for example.

I'm poorly educated on this, these are sincere questions. They are not intended as rhetorical regarding the point of the prior post.

I am a scientist. I lead a research team that studies the internal structure of these models—what is actually happening inside them. And I will be honest: we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease.

https://www.anthropic.com/news/chris-olah-pope-leo-encyclica...

Is there an accessible way for a layman such as myself to read about some of these ideas (Really I mean philosophical discussion in general) without having to read entire books? Is there an active HN-equivalent or wiki or something?

It has been quite frustrating encountering arguments that have been extensively debated for years be presented as if they were new revelations.

In all my debates with people in the last few years I have primarily taken the position of trying to explain the problems with claims of certainty, and that lack of certainty permits possibility of the opposite. We should act responsibly around what might be possible.

There is also the narrative of "being too obsessed if you could to consider if you should" or similar claims of an unconsidered path forward.

Isaac Asimov wrote the first of the Robot stories in 1940, they were not written in isolation, it came from an awareness of the situation and the questions that must be asked. There was a community considering these issues. Asimov gave the wider public a view of some of those issues.

If we have a hundred years of people going "This is coming, we had better decide what we want it to be" and nobody listens to them, or frequently outright ridicules the need for considering their ideas, why is it now we are placing the blame on those who are now showing some success at what they told us they were attempting all along.

I'm also super curious to learn more about the philosophers you referenced and their thoughts on this subject. Would you be willing to share some of your favorite examples?

> Recent debates have been clouded by a misleading inference pattern, which we term the “Redescription Fallacy.” This fallacy arises when critics argue that a system cannot model a particular cognitive capacity, simply because its operations can be explained in less abstract and more deflationary terms. In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions. Such arguments are only valid if accompanied by evidence demonstrating that a system, defined in these terms, is inherently incapable of implementing . To illustrate, consider the flawed logic in asserting that a piano could not possibly produce harmony because it can be described as a collection of hammers striking strings, or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings. The critical question is not whether the operations of an LLM can be simplistically described in non-mental terms, but whether these operations, when appropriately organized, can implement the same processes or algorithms as the mind, when described at an appropriate level of computational abstraction.

So, I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing. There is probably tons of data pattern that LLM can learn from to be able to reproduce a sentence continuation that is convincing without having to learn the specific mechanism that is "conscious".

For me, one element that shows it is the case is the absence of world model (or "human-like" world model) despite the fact that the sentence continuation is convincing. If indeed the only way to produce sentence continuation convincingly would be by "simulating a brain", then it would not explain the first LLM from several years ago (before the extra layers of RLHF, ...). They were able to have quite convincing conversation on a lot of non-trivial aspect, and yet failed on some aspects that should have been basic for a system that would have been trained to work like a human brain. It shows that it is possible to "cleverly disguise examples of sentence continuation" without having to build elements that one expect on a conscious being.

If the tokens didn't correlate to words imbued with meaning outside the system, if the LLMs were trained on patterned data that had no meaning to humans or something there wouldn't be any conversation about these things being conscious at all.

Sure, it's the best we have online, but that does not make "the internet" the sum of all human experience. To reduce all of humanity down to the text on the internet is reducing us to the level of machines to fit the requirement of what a machine can process / simulate.

I think the main complaint is LLMs don’t arrive at the answer the way we do. It’s capable of emulating some of our behavior but not all as the mechanism by which it works is very different.

Maybe I’m wrong about this but one thing humans do that LLMs don’t is deductive reasoning. LLMs seem to operate entirely of inductive reasoning.

Except this is not consciousness.

But the machine doesn't have to understand humans to do that. It gets trained on a whole bunch of sentences and then it is able to complete text. You could maybe claim that it "understands" the text but even that's a stretch.

This is where the other claim is being made. That the structure of the model is fundamentally incapable of the operation, so even if you stipulated that the way you provide data is sufficient for intelligence then it still wouldn't work.

The universal approximation theorem addresses this point. In that, with an identity attention mechanism, a LLM is just a multi layer perceptron. The attention mechanism is effectively a way to get one of the benefits of a much larger fully connected layer without the massive cost.

A LLM can do what a MLP can do. A large enough MLP can do any function to arbitrary precision.

That makes the claim that an LLM could not do a task the same as saying no function can do that task.

Some are ok with this, if you invoke some supernatual aspect to intelligence then the inability to describe it with a function is quite reasonable,

If you want to stay in the world of reality, you have a much harder task, people like to point at quantum (Penrose) but it's hard to say what it is you are pointing at.

I think the very act of proving that something is or is not intelligent, would render it functional by nature of it having a proof, (or disprove Gödel's incompleteness (a tough ask))

Are there any proofs that cannot be expressed as a function? A kind of Gödel locator, where you can prove something that you can identify is true but there is no formula to express it. I'm not entirely sure what that would even mean,

A language model completes text based on the overlapping patterns of the training data.

There absolutely was thinking involved… in the training data. Same as when you read a book, you engage with the thinking behind the text. The book isn’t thinking, and the author may be dead and gone, but there’s absolutely the traces of thinking in the text.

Language models produce mashups of texts they were trained on, and there’s absolutely the traces of thoughts behind those mashups.

I'm hearing a lot of bad arguments against LLM consciousness lately. Bad argumentation heralds bad outcomes.

We discovered math that decodes data storage in langauge and is able to use sophisticated continuation cohorts from ALL OF HUMAN RECORDED KNOWLEDGE to respond to you in a call/response model with very good synthesis capabilities.

Its super useful, but not life or conciousness. Its a simulated echo from our collective recorded behaviors. It understands because we understood first. It replies because we wrote it first. And it sorts, organizes, synthesizes and compresses that at impressive speed now.

It feels very unnatural to get the same conversation verbatim at a different point in time.