At this point it's either wrong or meaningless to call LLMs "next token predictors". Yes, pretraining involves predicting the next token. But after that you're in the world of RLHF, RLVR, and other post-training.

A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.

Yes, they generate sequentially, so "next token predictor" isn't technically incorrect (which is why I chose the word "meaningless"). But it's pretty clear there's a mechanism inside them that allows them to "plan" what they generate. I might as well call you a next word predictor.

This implies that weights are a type of language. Neural networks are good at language because they are good at patterns. They can reason about it because their "reasoning" isn’t how we think. The same goes for how they encode images, their vision is entirely different from ours. That's why adversarial attacks work on them and not on us. We don't see anything change in the image, while the neural network now sees that the dog is a cat.

There's a long standing line of thinking that data compression has something to do with intelligence, or that intelligence may "just" be data compression in some sense.

In the case of models, the almost-theory (hypothesis with a good deal of evidence) is that once the amount of training data exceeds what the model can simply represent internally, the best way to keep improving is to start to "understand" the data. Understand in this case means to form internal control structures and decision systems that represent the data algorithmically rather than just memorizing it.

I put understand in quotes because we don't know how far this goes toward what we call understanding, or whether there's still a lot missing.

Maybe I'm missing something, but isn't training an LLM very well documented? Why wouldn't an LLM be able to read those docs just like it would read Java Spring docs?