That's insane. How does that work? Does it take so much less data now, or smaller models, or what? Apparently GPT-2 cost ~$40,000 to train when it came out.

I'm sure Moore's Law for GPUs helped a little, but I'm really curious about the savings on the methodology side. (The repo mentions "hyperparameters")