Ok then, when my GPU runs No Man’s Sky, I don’t get confused and think it’s running a universe and that universe is real, nor that anything about that system is conscious. When I close the game and load the LLM, I still don’t think the same machine has a case for consciousness even though I think it’s super smart and helpful.

I think it's not really about having a conversation - I mean, that's part of it, but alone it's an illusion that eventually fades quickly. It's more of because of how it demonstrates intelligent behavior in reaction to requests, both in trivial and complex matter, and all across the board. LLM's response may be completely incorrect or confused, but it's nearly always exactly what you expect from a human[0]. This creates a more general feeling you're dealing with a human-like intelligence.

To be clear: I'm not talking about surface level things like prose. I'm saying that no matter what you do - whether you just paste a truncated log of a command into it with no further comment, or talk like a drunk teenager with no appreciation for grammar, or mix natural languages, or mix natural languages and JSON, or whatever else, the reaction you get is always that you would expect of a helpful person that got your message. It'll try - and usually succeed - to parse out what you actually meant, and deal well with subtleties around it.

This alone may not be enough to call it conscious or intelligent, but at the very least it's a large leap in that direction, and a qualitatively new functionality that classical software does not posses.

--

[0] - This is by design, not accident. "Respond to arbitrary input in a way that makes sense to humans" is literally the overall goal function the LLMs are trained to.

It is difficult when humour and trolling are forbidden. It was easier to tell Slashdot posters were conscious. We could easily reach the stage soon where your agent is responding to my agent and we just leave them to it to run HN automatically.