upvote
Beating even a very small LLM at next token prediction is basically impossible.

[2212.11281v2] Language models are better than humans at next-token prediction https://arxiv.org/abs/2212.11281v2

We knew this many years ago. Like GPT 1 era.

Actually you can play it here. The game is pretty frustrating ngl. And I can clearly see why induction heads are a thing. Because when a word repeats you guess is to just guess the same thing from before.

Language modelling game https://rr-lm-game.herokuapp.com/

reply
Will try this out, thanks
reply
Probably high because humans aren't trained on lowering cross entropy loss. Kind of funny to think that next token prediction is actually an emergent capability of the human brain...
reply
We are, though, but only partially. When listening and reading, we pretty clearly build a model of what we’re hearing or seeing and use it to correct for the rather noisy channels we use.

But when thinking, I’m fairly confident that our ability to predict the next few words of a book on a complex topic (in its words as opposed to the reader’s own words) is at best weakly correlated with intelligence or with subject-specific skill.

I certainly feel like I’ve learned a subject better when I can solve a problem differently from a solution I’ve seen before.

reply
I linked a forum post asking this exact question elsethread (https://www.lesswrong.com/posts/SZ3jDHXHb4WF4jmbr/where-is-h...), and the answer is frustratingly dependent on the context length, the dataset, and the knowledge of the person about the topic of that dataset (e.g. I would have high perplexity in American history).
reply
deleted
reply