Funny how you said "it turns out" when the whole point is that we don't understand LLMs - we just empirically see what they are good at.

Claiming that you understand LLMs is similar to saying that you understand how our biology work because you understand evolution. No - you understand the mechanism behind evolution, but not the complexity it produces.

This not understanding "understanding".

> maths are internally coherent and entirely theoretical

Nope. This kind of wish-washy thinking is not what we mean by understanding.

https://iep.utm.edu/math-inc/