Can you use a human language well enough and it not sound intelligent? And no, I don't mean just syntactically correct because semantics is a part of human languages too. I think the astounding thing is that they managed to model a language this well. It's like trying to learn English without a functional understanding of the word "Please".
My take is that a lot of the intelligence we natively ascribe to humans exist in human languages and just like the nutrients in your food, by harnessing a language that possesses latent intelligence, you too can behave like an intelligent person. And how did the nutrients of intellect got in there in the first place? Just like languages evolve nowadays: via contributions from speakers over time. Not unlike soil fertilisation.
It's funny, I used to think it was complete and utter horseshit when literature professors would say that the meaning of a text is created by the reader, not the author.
Since LLMs have become commonplace, I have completely reversed my opinion on that.
That you see recognizable intelligence in the output of stochastic generation is because you are imposing meaning on it. It is why "claudish" is so hard to read--you are trying to impose meaning on a sequence of words that was stochastically generated, and it takes mental effort to do that.
There's pretty interesting (I think) research in how effectively it's possible to scale down LLMs. If you're interested, take a look at the BabyLM conferences:
(I have no connection to them.) The work there demonstrates that gap between what we can do now with human-relevant amounts of training text, and what remains to be discovered. Quite a bit, it looks like.