It's the thing that minimizes the loss during the RLHF phase, and the RLHF phase is the one that's aimed at maximizing engagement (it's literally trained on that).
> what happens to the actual humans whose writing style is a close match for what a given generation of LLMs output?
If a human, for instance because its writing gets polluted by reading too much AI slop, matches the style of an LLM closer than a certain threshold, then his own writing is going to be flagged as well. Whether it's an actual problem or merely a theoretical one is an open question. (unlike OpenAI and Anthropic, humans writers do have an incentive to avoid being flagged as AI).
> And, what stops LLMs from using a different style when someone wants to fool the classifier?
In theory: nothing. In practice if you fine-tune your own model: nothing. In practice with commercial models: the interests of the model making company.
> And, what stops LLMs from using a different style when someone wants to fool the classifier?
Websites have pretty much stopped using ad-blocker-blockers, it seems that it's not a fight worth fighting for them. Does that mean that ad-blockers are useless?
Most people don't even care about ads, I don't think they care about slop either, that's why there's slop posts and obnoxious websites that are unreadable without an ad blocker. A slop blocker used by 10-20% of the internet users wouldn't change the calculation more than ad blockers did.