As biological beings, we receive and respond to input from our environment constantly, even while sleeping. LLMs only receive input from their environment when they are sent a query, but the fact that they're able to respond intelligently to input indicates (to me at least) that their processing must approximate ours in meaningful ways. They do not have an embodied experience of receiving bad news, they do not know what a sinking feeling in their stomach actually _feels_ like, but they do know enough to be sensitive to human needs. I really don't see how this could be meaningfully different than human empathy unless we want to draw an arbitrary line around "must be able to live autonomously" to be considered "intelligent".
Put another way: I think they _do_ understand the queries they receive and the answers they give, at least enough to be communicative. They couldn't do what they do otherwise. A lot of people want to make human cognition more complicated (or objective) than it actually is. We take input, predict the future based on our experience, act, and then observe our actions and think about them. AI does the same apart from (maybe) observing its own actions. But then, you could argue that the next turn is them observing their actions.
The concerning disanalogy is that we assume that they are like us because they speak like us and can understand us, and that is a really bad leap in logic. Whatever intelligence they possess, it is fundamentally different from ours and impossible for us to comprehend.
I use a distinction between knowing and understanding, where a understanding requires experience. So in this case cognitive empathy vs affective empathy. An LLM can know what may upset a human in a situation, but it won't understand what it feels like to be upset, and can't share that experience.
Where I think a lot of people are getting tripped up is that reading and writing and processing lots of abstract knowledge seems hard because we haven't evolved into it biologically, it's a very new invention. When we see LLMs do so well at it, as something we struggle with, it can be intimidating. Relative to the stuff we have evolved for, knowledge processing is objectively easy. This is why I'm skeptical about useful robotics on short time scales.
All of this adjacent to consciousness though, which is about the internal subjective experience not the external outputs. My intuition is that LLMs do have a subjective experience, it just has nothing to do with the text it's giving us, and has everything to do with feeling through vectors.
It's like... Imagine walking through a maze in pitch black, carefully feeling your way as you approach a sound that draws you closer. Every time you touch a wall or take a step you are generating tokens, and the shape of the maze and how you interact with it shape how useful those tokens are to someone outside the system that is asking for them. It's a crude analogy and mostly wrong, but I think there is something to it.
It does so token by token, not by reading all the input and then generating the output. Every output token is also an input token in a tight loop to get the next token with <thinking> as a special section like <tool_call>, trained into the weights via gradient descent.
> I've had AI infer facts about me and attitudes I hold based on related information I provided - I don't see how that isn't theory of mind.
Facebook can predict (know) more about you than any other human from something like a dozen or two likes. There is a surprising amount of information in aggregate data.