Though of course you're talking about data centers, and romanticizing them rather than the AI itself.
There are various theories around model collapse when you train on too much AI generated data (that's not for distillation).
The summary of this paper describes my sentiment in better words than I have:
https://www.nature.com/articles/s41599-025-05868-8
It’s very easy for the average person to mistake linguistic ability and simulated problem solving for intelligence and sentience.
I do not believe current AI or LLMs are conscious, but there is no proof one way or another that they can or cannot be. The paper authors are making up their own definitions and building an argument from them
I, and apparently many others, don’t think it would be any useful to describe the mathematical properties of an AI as consciousness. To me it is inherently a way to describe the “experience” arising from physical processes in biological beings as ourselves.
That’s what the argument comes down to for me. Could an LLM “fall unconscious”?