Yes, I agree with that. Consciousness is a good way of generating convincing human text.
What I don't agree with is that consciousness is the only way to generate convincing human text and that because we have convincing human text, it can only imply we have consciousness.
There is a huge probability that generating convincing human text can be done without consciousness. Either because there are efficient mechanisms as efficient as the way the human brain deal with this problem and that the LLM found one of them (and these mechanism may be quite difficult to imagine for a human). Or even because the LLM found a local minimum and is stuck there.
To re-use the evolution approach: evolution solved the "flying problem" with bird feathers, but also with insect wings or bat wings. The fact that evolution ended up using feather does not imply that everything that flies can only fly with feathers.
> World models don't have to be rich and detailed to count as a world model
I agree in general, but here, we are talking about machine that reproduce all human language. The argument I'm answering to is pretending that "all of human knowledge" is understood, which include every single human concept. This has to be everything, because LLM is able to provide convincing text about every subject. If on some subject, the LLM is able to provide convincing text without "understanding" it, then the argument that it is impossible to provide convincing text without understanding it collapse.
> There is nothing that a priori constrains token prediction from the domain of consciousness.
We don’t know either of these are true or false though. We simply don’t know. There is no agreed upon definition of consciousness, aside from maybe _the having of qualia_, so arguing that some can or cannot be conscious a priori can’t be done.
No one genuinely engaged with the topic is confused about the target of the term (phenomenal) consciousness. Definitions come once the theoretical work is complete, to be articulated as part of a fully worked out theory. The lack of a definition doesn't prevent us from investigating the subject or offering conjectures. What we can do is offer a precise description of the target and argue for or against whether LLMs reach the description. We will of course debate whether the offered description captures the relevant phenomena. But this is all just part of the process.
Try to define qualia though without explicitly or implicitly recursing into consciousness.
It's all a large house of cards that's built on handwaving and "I know it when I see it".