Ok so the correct comparison would be to fix the harness on the old model and re-compare. Now they are comparing a new model to an old crippled one.

Exactly what I suspected. Of course a machine can just iterate relentlessly the way a human can't.

I guess token counts are somewhat of a metric.

IMO intelligence has peaked and all future gains will come from faster tps and more iteration.