upvote
Almost verbatim addressed in the post under “Perplexity as a shared metric has challenges”
reply
This metric is purely scoring the ""intelligence"" of the base model, not including RLVR/chain of thought capabilities. It's widely believed that the pretrain determines the ceiling that RLVR can take you to.
reply
I think it can still be a useful metric, but probably isn't the "one true" metric. The trend is pretty clear, and the acceleration of capabilities since December 2025 definitely aligns with my subjective experience.

There are probably some other axes where o3 is clearly better than o1, but with how complex intelligence is no metric will be perfect.

reply