Deepmind was proving math theorems using a model specifically trained on math, and that was very quickly surpassed by generalist LLMs. It's presently not well understood why this is the case.
I am not simply talking about applying a mathematical technique commonly used in one field to another. At a deeper level, you always get back to playing the same game in a different form. For example, sales engineering has a lot in common with cybersecurity.
And this is a good example, as it's extremely rare to find someone with advanced expertise in both fields, since they typically attract two radically different kind of people.
For this reason, where the output expected is not mechanical, i'm not surprised at all that generalist models are doing better than specialized ones for open ended questions and frontier work.
If pretraining alone was sufficient then nobody would do RL. And, heavy RLVR on coding/math is why these models are savant-level in those domains, yet not superhuman at all fuzzy reasoning and practical wisdom tasks that are only represented in the pretraining corpus, and don’t have an RL target.
In math at least I often use analogies to help solve problems, like "ah maybe if I think of this as a vector I might use linear algebra here" and such.
I could believe that your RL/test-time scaling curves are strongly constrained by the pretraining loss and so in some sense that makes the max/optimal performance “latent”.
But its not clear that pretraining loss alone gives enough information on capabilities, particularly as we get closer to dangerous thresholds in some dimensions. (Eg CBRN capabilities, to take a mundane harm that’s currently impacting us directly.)
I’d like to see the scaling curves here; intuitively I’d guess you get more return from RL/test time, the smarter the base pretrained checkpoint is; therefore the error bars on capability/perplexity should be getting wider on this chart.
But there’s definitely spikiness and it’s very possible they’ve trained more on math than other domains.