points
I had the same thought and just started writing something about how maybe they just added more code to pretraining corpus over time.
Edit: here it is https://micjm.com/blog/lowering-perplexity-without-scaling/