upvote
We have scaled on both axes and many others, I'd say.

Inference can be increased vertically (a single model thinking for longer), or horizontally (NS was solved by a 10,000-agent swarm).

And pretraining has two inputs that can be scaled independently - training data and model size. (Things like the Chinchilla 20:1 law are meant to establish the optimal way to train with a limited budget - you could just keep making either term bigger and bigger on its own if you had infinite money.)

I think creating and curating high-quality datasets is also part of scaling. Datasets like FineWeb are quite a bit better than the barely-filtered CommonCrawl dumps GPT-3 trained on, but someone had to invest in making them. (To say nothing of whatever exotic bespoke robotics/physics/etc datasets OpenAI and Anthropic are investing in.)

reply
ChatGPT was only the turning point for public awareness (and the massive influx of institutional investment that followed), but researchers like Ilya, Alec Radford, and Dario were already convinced years before ChatGPT that scaling compute was the way forward. ChatGPT just helped unlock the resources (by proving the commercial potential to investors and the subsequent influx of capital) for these guys to pursue that vision at a much larger scale.
reply
I mean, why wouldn't that be the default assumption, if you know anything about computing history? ChatGPT's (v1) contribution wasn't proving commercial potential but demonstrating efficient scaling, due to a confluence of technical developments. If you're talking about v3 then sure.
reply
> ChatGPT was the turning point of realizing we needed just more compute

Ilya Sutskever has been saying more compute = better perf since at least 2012.

  ... the real conclusion now is that if you have a large big data set and you train a very large deep neural network success is guaranteed ...
https://www.youtube.com/watch?v=-uyXE7dY5H0&t=1185

  In the end, the network's size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate ... All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets ...
https://proceedings.neurips.cc/paper_files/paper/2012/file/c...
reply