I don't think it's true at all, and I think we have indication that proves it is false.

We have "basic" LLM, the ones from 2023. They were producing _very convincing_ human text, and yet, they were too often failing basic tests that require understanding.

Now, we have more advanced models, but the counter-example of "basic" LLM demonstrates your assertion is incorrect: these model _did_ produce very convincing human text and yet did not make sense out of it.

But for the more advanced models, the problem is that they are "on top" of basic LLM. So, the first step is a training that build a mechanism that produce convincing text without understanding, and then, the "residuals" are fine-tuned. The result is very unlikely to add "understanding" to the model, because to do so, the whole system needs to deconstruct the basic LLM, to go back towards less efficient situations in order to rebuild almost from scratch. The fact that modern LLM are based on basic LLM means that the first step put the cursor in the bottom of the "basic LLM mechanism" valley, which is a local minimum. And any layer on top of it cannot "climb up" the slope of the valley, pass the ridge and fall into the next valley, even if this next valley has a lower minimum.

> The number of parameters models they have may seem large, but they are very small relative to the training data that they have to summarize.

That is demonstrably an incorrect logic jump. For example, CNN are able to distinguish between pictures of cats and pictures of dogs. The weights in these models are very small relative to the number of pixels they have been trained on. Yet, they distinguish cats and dogs by finding specific shapes in the pictures, without understanding what a 3-D cat and a 3-D dog is.

They have done that without discovering the typical human pattern that make sense of "cat" and "dog". And yet, the number of weights is very very small with respect to the number of pixel used in training.

> And we can verify that. Simply discuss completely disparate topics, ... > If the model is only interpolating it will produce gibberish.

What you are saying is that the model is not simplistic interpolation. But that is a straw man argument: people who say that LLM don't understand don't say LLM are equivalent to simple interpolation machine.

But the problem is that you can have very good predictions in novel situations without understanding.

For example, if you have 10 totally different situations that can be described with a Gaussian curve, and that I show you points for a new situations that cover the left side of a Gaussian curve. Then you will be able to guess that the right side of the curve, which is not an interpolation as it corresponds to situations you never saw, will behave like the rest of the Gaussian curve. And yet, in these 11 situations, I did not even say which real physical phenomenon I'm talking about. You haven't understood anything about these phenomenon, all you have done is guessed that a typical pattern that you have observed somewhere else is more likely to apply here too, without even having to understand anything about the reality of this situation.

And of course, this prediction is "a guess": maybe, for once, in this 11th situation, the curve will start as a Gaussian curve but will suddenly be different. But it happens that the reality is that in this 11th situation, the correct description is a Gaussian curve (because, due to the maths, Gaussian curves are really common). So, when you make your prediction, it looks like you understand the situation, it looks like you understood the physical mechanism that applied here. But it is not the case.

So, no, correctly doing such prediction does not demonstrate understanding.

> The fact that models can be near expert, and sometimes expert, across vast areas of human knowledge is a clue.

That is not at all sufficient. A Chinese room experiment will do that despite the system not understanding Chinese. A pocket calculator will be able to be expert in math computation.

> If they don't understand that, then the question is, why do we think people understand things.

That's the wrong question. The correct question is: we know people understand things, and we see AI behaving similarly to people in some aspect, but is this behaviour _requires_ understanding, or can we reproduce this behaviour without needing to understand?

The fact that "basic" LLM were able to reproduce very convincing text that look like they understood X and yet were demonstrably showing lack of understanding of X demonstrates that we cannot just jump to the conclusion that just because it looks the same, the only possibility is that the core mechanism is identical.

Firstly, how do you know that the optimal way to highly compress complex information is to understand it? You think it is obvious because you are very familiar with "understanding" as a way to summarise complex information. But there can be billions of different ways, outside of human imagination, that is as good or even better.

But secondly, LLM don't find the optimal way, they find the local minimum. Everyone who worked with NN knows that they are prone to come up with spurious pattern, incorrect correlations and bad workaround to guess the correct answer. You regularly need to nudge the NN by creating specifically engineered features to avoid them to fall into the first local minimum.

When it comes to LLM, it is extremely complicated to control to see if the LLM has triggered on a misleading pattern that, by chance, links two "tokens" together, or on a real concept that indeed links two "tokens" together. Basic probability implies that there are probably tons of "fake patterns" engraved into the weight during the LLM training, "fake patterns" that should not exist if there was any kind of "understanding" of the abstract mechanism that links these tokens.

1) Good current generation AIs are specifically trained to reduce hallucinations. If we had new AI system that happened to not have hallucinations as a side effect of their training, then it would be convincing. But here, it looks like we have built a pocket calculator that answer 7+13 = 14, and on top of it, we added a layer that says "if the input is 7+13, then replace the output by 20". This pocket calculator still does not know how to calculate, we just added a layer to hide its mistakes.

2) Not only "make shit up" is not the same as "hallucination" (either "making shit it" is done when the individual knows it is unreliable, or when the individual was given wrong inputs), but the point is not to say "hallucination implies no consciousness", but "large quantities of hallucinations in situations where a conscious system would be unlikely to hallucinate implies no consciousness"