upvote
Off the top of my head: lowering perplexity on human-dominant data might mean that we're leaving behind "the model is innately inept and can't predict some things well" and getting closer to "the model is innately capable, but knows human idiosyncrasies and is holding its full capabilities back when predicting human texts."

Thus, if RL-trained for pure performance/correctness, the "innately inept" model would typically underperform the "innately capable" one. The second model has started from a better foundation, so when trained to push itself past the usual limits of human-dominated data, it would attain better task performance.

reply
If the task is to predict what a human would say, then a perfect oracle would produce what a human would say, even if it was factually incorrect.
reply
But if the task is to be correct and useful in the maximum likelihood then this is unlikely to be the metric used
reply