upvote
I work at OpenAI. The correlation between perplexity and the quality isn’t 1, but neither is it 0. It makes sense that as a model gets better at learning pretraining data, it also gets better at generically learning whatever you’re trying to teach it afterwards. The relationship is imperfect and empirical, and if it breaks down in the future, we’ll of course adapt to different measures instead of dogmatically clinging to the old paradigm. But so far, it’s still informative.
reply
I think his point (unless i've misunderstood) is moreso if a codebase is really the best held out dataset for such a evaluation meant to be wide-sweeping. Does this plot look any or much different if the dataset was more encompassing to include other sorts of text ?
reply