>how have you concluded that it’s certainly not an emergent property of something like a highly trained LLM?

the same way I (and likely you) have concluded it for anything else. We don't assume objects that share no similarity with human or animal physiology or evolutionary development are conscious (let alone happy or anxious). 'Emergence' isn't a word you can abuse to justify your a priori assumptions in the absence of an explanation or even reason to assume something exists.

We can say a chemical property emerges from the configuration of a molecule because we can explain the process by which it does and observe the property, when people claim that consciousness "emerges" from an LLM they posit that it is conscious, and use "emergence" as a gap-filler to explain away the need for the process by which that allegedly occurs.

If you want to know the neurophysiology or evolutionary biology of pain or anxiety, which we do know quite well you can find them in a textbook, but suffice to say transformer models don't share any of them.

And importantly if anyone seriously believed transformer models were capable of conscious experience as Chiang points out they would have been in despair when AlphaFold was released, it's structurally a virtually identical system. But nobody did, because it didn't 'talk to them' through a chat interface.