Off the top of my head: lowering perplexity on human-dominant data might mean that we're leaving behind "the model is innately inept and can't predict some things well" and getting closer to "the model is innately capable, but knows human idiosyncrasies and is holding its full capabilities back when predicting human texts."
Thus, if RL-trained for pure performance/correctness, the "innately inept" model would typically underperform the "innately capable" one. The second model has started from a better foundation, so when trained to push itself past the usual limits of human-dominated data, it would attain better task performance.