It's more of a vibey term, and as such it is genuinely very difficult (perhaps impossible even) to concretely determine whether an LLM possesses consciousness. LLMs successfully express a lot of consciousness-like traits.
At some point you have to ask the question: does it even matter? If an LLM can sufficiently mimic consciousness, isn't that sufficient for us to treat it as conscious, even if it is in-fact not conscious (especially because we don't actually know)?
- Are current LLMs conscious?
- Is it possible that future versions of LLMs with similar architectures could be conscious?
- Can any AI be conscious?
If it was true, you can create extreme pain by running a program. You can run the program by simulating a CPU, using pen and paper for memory. So you're essentially claiming that some simulated being is in pain because there are some 1s and 0s on paper. In fact, you can decide to use an arbitrary encoding of the memory, so a sufficiently long sequence of 0s written on paper corresponds to a simulated being feeling pain in some encoding. That is clearly nonsense.
The crux if it is that if you ever break from "the universe can be fully expressed mathematically", you are stuck in the mud of supernatural beliefs.
I try to imagine myself long ago, on the outside looking in, with someone explaining to me that extreme pain, wondrous art, hunger, triumph, and despair would all unfold in due time where the rocks were wet and the lights bright enough.
I can imagine myself calling this clear nonsense.
If you don't understand that, how can you assert that it doesn't come from mathematical relationships?
If you would have zero knowledge about ICEs, how would you know?
The only strong argument I have against it is the anthropic principle -- there are billions of times more bacteria than humans, so it's overwhelmingly unlikely that I'd be a human rather than a bacteria.
Not a very good argument of course.
Pretty much every single idea in science can be traced back to some combination of earlier ideas, and as they get earlier / simpler, they can be related back to some combination of direct observations.
It's not clear to me at all that our entire body of scientific knowledge can't be simply recreated by "combining results of observations + previous information". And LLMs can perform observations in addition to combine previous information, which in my estimate is genuinely sufficient for them to plausibly be able to rebuild all of science.
If every piece of knowledge is created by combining previous ones, then no true grounded knowledge is possible. Every bit of knowledge is grounded by another bit of knowledge which is itself grounded on, ..., etc, etc.
"If every piece of knowledge is created by combining previous ones, then no true grounded knowledge is possible" - this is not true, and has a very simple solution. What you are calling "grounded" knowledge comes simply from observation.
When the very first human saw that an apple falls of a tree, they have acquired what you call "grounded" knowledge. They can then combine that knowledge with the grounded knowledge that other things fall from trees to start reasoning about why things might fall from trees.
The problem of induction is different. It says, we know that things fall from trees because things have always fallen from trees, but how can be certain that what we call the laws of physics will not arbitrarily change on us?
And of course, it's a famous problem because it doesn't have a satisfying answer. The answer is basically "well everything we know is wrong if induction is wrong, so we will pretend induction is not wrong and hope for the best". And at least so far, that approach has seemed to work (heh heh heh).
The funniest thing is that LLMs will lap people in those capacities way before people who think like that accept that they might be conscious.