It makes sense though. How often have we heard of people having insights while they're in the shower, or in a dream? Obviously our brain is doing a lot of processing on problems when we're not consciously thinking about them. I think most people that do deep work probably find that they can intuit a solution before they can really verbally explain it
replyReasoning ability has to be independent of consciousness because there are different levels of intellectual ability among people but everyone is equally conscious.
replyChiang is very right in saying that the lesson of LLMs isn't that LLMs are conscious, but that there are broad areas of reasoning that apparently don't require consciousness.
We don't have anything to measure consciousness generally. We don't even have a broad consensus on what consciousness is. And because of that we can't discern whether X is conscious for most values of X, including but not limited to LLMs.
replyWell, at the very least we have a base to distinct different kinds then.
replyProbably the main problem of people implying LLM consciousness is that they imply LLM have human-kind of consciousness. Judging only on "how they speak", generally, they insist on using the same word that labels human consciousness (exclusively), etc.
But there are so many instrinic differences that such claim is not feasible despite similar "talking abilities".
Ai separates language and reasoning from humans, like writing separated memory from us
reply"AI does to us what American Cheese did to food" ~ Opus 23