So most training data would be grey and a little bit coloured? Ok, that sounds plausible. But then maybe they tried and the current models get it already right 99.99% of the time, so observing any improvement is very hard.
replyThey have a lot of data in the form: user input, LLM output.
Then the model learns what the previous LLM models produced, with all their flaws. The core LLM premise is that it learns from all available human text.
replyThis hasn't been the full story for years now. All SOTA models are strongly post-trained with reinforcement learning to improve performance on specific problems and interaction patterns.
replyThe vast majority of this training data is generated synthetically.