If the prompt is unique, it is not in the training data. True for basically every prompt. So how is this probability calculated?
Type "owejdpowejdojweodmwepiodnoiwendoinw welidn owindoiwendo nwoeidnweoind oiwnedoin" into ChatGPT and the response is "The text you sent appears to be random or corrupted and doesn’t form a clear question." because the prompt doesnt correlate to training data.
But the human brain (or any other intelligent brain) does not work by generating probability distribution of the next word. Even beings that does not have a language can think and act intelligent.
Wait what? So a robot who is accurately copying the actions of an intelligent human, is intelligent?
If it's just basically being a puppet, then no. You tell me what claude code is more like, a puppet, or a person?
But that is the key insight, how can you tell when an imitation of intelligence becomes the real thing?
If the idea is that something cannot accurately replicate the entirety of intelligence without being intelligent itself, then perhaps. But that isn't really what people talk about with LLMs given their obvious limitations.