Wouldn’t that be extremely computationaly expensive considering how resource incentive training is?
replyNo, training a state of the art model involves training on the order of 10 trillion tokens.
replyWe're talking about a step that updates weights based on say between 10k and 1M tokens.
I learned something. Thank you!
reply