I guess public datasets on HuggingFace and some shadow libraries content is enough to start.
replye.g. fineweb dataset is 50TB https://huggingface.co/datasets/HuggingFaceFW/fineweb
There are a lot of open-research on pre-training, post-training and RL data mixtures and sourcing.
replyI recommend checking papers from Datalogy, Nvidia Nemotron, Ai2 (Ollmo, Tulu, ...) and the recent model from Aleph Alpha if you want to learn more.
If you ask a model, they will generally tell you where to get data. Modern frontier models have the large advantage of having tens if not hundreds of millions of users providing use cases to train against to improve their responses.
replyhttps://scale.com/data-engine - you just buy it.
replyforget the data....sell it and go live your life!
replyGet data from Claude. That's what the Chinese (allegedly) do.
replyNote that this sort of distillation is NOT for pre-training data (which is tens of trillions of tokens). I think the allegations against Chinese companies by Anthropic is more so that they distill SFT data (which is good for post-training, but you still need a strong base model)
reply