Not GP, but one of the challenges with debating whether LLMs are "conscious" is that we don't even really know what it means for a human to be "conscious", or even if consciousness is experienced by other humans the same way it is for ourselves.

What we do know: neurons carry electrical impulses across their synapses to trigger other neurons to fire, and more frequently used synapses are strengthened while infrequently used ones are pruned. This is not all that dissimilar to how a multi-layer perceptron is trained: it's floating point numbers in a big matrix rather than biological structures and electrical impulses, but there is still that element of frequently used connections being strengthened and infrequently used ones being pruned.

What we hypothesize but do not know: there is a thin brain structure of grey matter called the claustrum that has tendrils that reach into nearly every other brain structure. In many ways, this is similar to the attention mechanism of the transformer architecture. It is hypothesized that this may be the seat of consciousness, owing to experiments where electrical stimulation of the claustrum caused patients to immediately lose consciousness. However, there is no way to prove this, owing to the difficulty of otherwise removing or disabling the most connected structure in the brain and observing its effect on consciousness without permanently killing the patient.

Beyond that, we don't know much. I've got a family friend that's been a practicing therapist for 50 years, and I asked him what was the most interesting observation he made in his career. It was that "Everybody experiences the world in a different way, and yet everybody assumes that everyone else experiences the world the same way they do."

I believe you should look up the work of Cameron Berg before making statements like "an LLM is" or "an LLM isn't". Empirically defining all this stuff is very difficult, and making a definition that covers all beings that can exhibit conscious behavior is much more complex than a face value examination would reveal.

You are saying the AI doesn't a some property that you don't have any definition for, not even a working definition. People will disagree on whether a cat or a baby is conscious, they're not debating what a baby or cat is. They're debating this term. You might as well be debating whether an AI is a blorb or not, you have just as good a working definition of blorb as consciousness.