We absolutely don't know enough about LLMs and intelligence to make such a bold (and ridiculous) claim. If anything, all evidence point to the contrary, with new scientific breakthrough achieved across a variety of fields via LLMs.
I've been really struggling to understand how the HN community can so boldly claim that LLMs are going to stop improving or not really smart. I just read it as the "denial" stage of the stages of grief that a good portion of this community is in right now (which is understandable).
We know exactly how attention layers work and how they produce the next word as well as draw them from larger feature spaces.
If we would know that, there would be no need for interpretability research.
I tried to generate the next word to the best of my ability, starting with a mathematical problem, but I did not create a valid proof. How do these LLMs work when they create math proofs to problems not yet solved?
> maths are internally coherent and entirely theoretical
Nope. This kind of wish-washy thinking is not what we mean by understanding.
This is like saying we don‘t know how a car works because a car can beat the best human athletes in 100 meter dash.