upvote
I thought that too, but models are trained on observation now, to increase agent performance. This is new data that is separate from Internet crawls. I think it's done with reinforcement learning with verifiable rewards

https://github.com/opendilab/awesome-RLVR

Examples:

- When playing Go or chess, nobody may have written or published web pages with a particularly clever end game, but an agent can use CPU time to discover these games themselves, and learn from them. Human Go players report learning strategy from what AlphaGo discovered.

- Agents use Unix shell and are trained with it. So they can discover things and combine things that nobody has written about on the Internet, like how to debug a rarely used driver, or an interaction between two drivers. And they can see patterns in all the drivers that humans may have a hard time even seeing, leaving aside the motivation to publish on the Internet.

This explains why the performance is so "jagged" - great on coding, math and closed games, but less good on things involving the real world. We probably did hit a data wall, but then we got new sources of data.

reply
An interesting experiment would be to create a transformer model that is just designed to play Chess or Go. Specifically, the only thing that should change is the number of tokens, consisting of:

- the row of the move;

- the column of the move;

- the piece moved;

- any special/additional states from that move -- check, checkmate, castle, etc.;

- maybe an indication of whose turn it is (i.e. a black piece and a white piece token).

You could then train the base model on published game transcripts and use the interactive RL approach like AlphaZero and LeelaChess use, as well as what you mention for LLMs.

Why this is interesting is that it should give insights into how LLMs/transformers learn Chess/Go without having other data affecting the results. Specifically whether the token semantics identify valid moves, etc. and whether the context window encodes gameplay strategies across the different layers.

reply
Yes, they started to hit a data wall, but I thought that much of the progress of the last couple years came from reinforcement learning after pretraining, which is mostly building complex environments for the models to reason in, rather than accumulating piles of raw text.
reply
I'd hope they measured PPL on the same dataset for all those models. And one that's fair and doesn't benefit some models more than others (probably non synthetic dataset).
reply