Modern pretraining also consists of expensive human-led specialized task creation and grading loops. Synthetic generation and distillation from previous models is another input for training. I wonder how much new text contributes beyond keeping knowledge up-to-date.

The measure is to see if the results scale, not just the rumored attempts at building such a model, as o3 taught us.

Where are they getting new data from at this point? Didn't they already read the entire internet?