> This is what LLMs do. They're generating meaningful, coherent text

No, they generate grammatically coherent text. That is because human language grammars are fundamentally mathematical structures that can be approximated with matrix operations.

They don't generate meaningful text because they have no inherent knowledge of the world.

If you've used LLMs for any amount of time you've already noticed how often they get confused about numeric quantities - like confusing notions of "bigger than" and "less than" or being unable to count letters in words.

This is because any meaning in their output is only accidental.