I sort of agree, but within the same model I expect the reasoning effort to be reflected in the quality of output and that's basically how it played out. When you're comparing different models, then it's just who benchmaxxed the best and there's not a lot of value there.
replyBut that is my point: if benchmaxxing was all the labs were doing, then surely the dumber model could/would have equivalent performance? Rather than noticeably worse perf on a (somewhat trivial to game) test.
reply