upvote
Pre-training these days includes datasets with full session rollouts with tool-calls, user/assistant turns, ...
reply
Sure, and learning to predict those tokens is probably a good way to start training a model. But the hope is also to eventually make a model that's _better_ than what's in that data, right?
reply