I don't think this is the issue. Most people are mediocre mathematicians, too, yet the companies have selected well enough for math literacy in training data (presumably). I think its a matter of training - they are often post trained by other agents (which read everything), human reviewers not paid enough to care (easily wowed by 10 paragraph length essay with correct punctuation), and deterministic scripts (checking for presence of a specific point, so agents are incentivised to write as much as possible to have more chances of hitting the unknown thing the script is looking for)
replyAs far as I know labs try to train the other way, else we'd have monstrously long and useless outputs, but its hard to overcome these natural biases for length
I don't think that's a suitable mental model of how LLMs work. Their output is not just the statistical average of all the training data. If it was then the output would be quite useless.
replyAn LLM trained on the entire internet incorporates essentially all (human-known) aspects of things.
replyLike a superposition of all kinds of writing styles, all perspectives on every topic and all kinds of ways to do things right and wrong.
In post-training (and of course already by sensibly curating the input data in the first place), people try to select useful submanifolds from that.
Ideally, you get a combination of all the extant strengths of people combined into one model. Self-improvement then leaves the sphere of human capability for good.