"Is X conscious or not" is an entirely unserious question today, unless this is just a headline followed by actual (and explicit) examination of the various aspects of consciousness being discussed. But, even still, LLMs are probably only conscious in like 2-3 or so ways, most of those meaning broadly "intelligence", i.e. reasoning, problem-solving, etc. When it comes to anything experiential or embodied, AI might eventually get more of these, but LLMs based on recursively applied linear algebra are clearly missing too many core aspects of consciousness to be considered conscious in any broad sense.
To quote wikipedia:
> It has been the topic of extensive explanations, analyses, and debate among philosophers, scientists, and theologians for millennia. There is no consensus on what exactly needs to be studied, or whether consciousness can be considered a scientific concept.
Move to the different aspects / parts / things involved when we talk about consciousness (experience / phenomenal consciousness, self-modeling, intelligence, agency, embodiment, wakefulness/alertness, attention, etc) and you can have very clear, meaningful, and unambiguous discussions on almost every point, but there is no coherent unified "consciousness" as normal people use it, and the folk concept can't be salvaged.
This article is bad because it just keeps trying to make the folk concept do work that the concept is simply too messy to handle usefully. But in fact if you avoid trying to find some mysterious essence or all-capturing definition, there is huge progress and lots of interesting stuff to say (https://plato.stanford.edu/entries/consciousness/).
This means that consciousness is fundamentally subjective and outside the scope of physics and science. That's why physics / science will always struggle to deal with consciousness. In order to understand consciousness, you need to make a huge paradigm shift, that there's something outside of science.
Consciousness can be thought as a window through which we observe the world and we use science to summarize patterns in our observations. But science can't explain or even define the window. Everything in science eventually boils down to subjective observations / perceptions, e.g. we see (subjective perception) that when we drop an apple, it falls.
The meta issue here is that mostly the online debate on this is a bit lazy and hand wavy. I'm not really up to speed with most of that literature so I'm not not that qualified to add to it. But I know enough to recognize when others aren't either.
For a proper debate, you'd want people that are expressing views that are at least grounded in the existing views or counter them in a way that holds up to scrutiny. This article doesn't do that.
As you said even just outlining what particular notion of consciousness the author subscribes to would be helpful. Which of course he doesn't and makes this article a bit of a sand castle based on a very loose foundation. There's a whole lot of "if this is true and if that analogy holds then this also needs to be true" that you could easily challenge. That all makes the article a bit of a nothing burger in terms of conclusions.
I happen to agree with the conclusion that AIs are not conscious. Yet. They could be. I don't see why not.
I suspect that if you attempt to rigorously define consciousness all the way down without handwaving, you might discover that it doesn't exist after all, or just decompose it into low-level abstractions while having the original meaning slip away (which is the same).
You may also want to look at functional equivalence analogies provided by mechinterp and functional anatomy of large models (not necessarily language ones). Evolutionary analogies as well.
The only purpose that can really be served by arguing "the LLM system is conscious, you see" is to prop up continuations like "... and therefore, it would be immoral to terminate this running process" (or expose it to radical political content, or ask it to analyze photos from a murder scene, or...)
How to measure that, or verify it, is the hard part.
(LLMs carry other numerous similarities to dreams or to certain psychiatric disorders. So there is indeed a mechanism in our brains that is similar to how they work. But it is not the only thing there and on its own it won't "evolve" into consciousness. Even if we believe consciousness evolved somehow, it would be hard to imagine it started as a delirious state and then somehow ceased to be delirious.)