GPT-6 is supposed to be using a much larger base model that just finished pretraining so the "dump another chunk of all written language" approach is still going strong.
replyModern pretraining also consists of expensive human-led specialized task creation and grading loops. Synthetic generation and distillation from previous models is another input for training. I wonder how much new text contributes beyond keeping knowledge up-to-date.
replyThe measure is to see if the results scale, not just the rumored attempts at building such a model, as o3 taught us.
replyWhere are they getting new data from at this point? Didn't they already read the entire internet?
replyCustom sets and mining their own users, as every lab does
replyThe bitter lesson just means “compute scaling beats hand-tuned architectures in the long run”.
replyAs GP said. More RLHF is in fact the bitter lesson.