Yeah that's pretty much what gwern argues here[0]. Or to adapt another proverb: to predict the next token you first need to model the universe.
reply[0] https://gwern.net/scaling-hypothesis#gwern-difference--effic...
> to predict the next token you first need to model the universe
replyExactly. The "most likely next" series of tokens, for example, when given the first half of a correct mathematical proof, is the correct rest of the proof. I have never seen anyone define "most likely next token" in such a way that this isn't true.