> not sure it's apples to apples comparison.
replyThey're literally comparing the previous version of the same model with the new one. It's based on the same architecture, same pre-trained model, just different post-training. It doesn't get more apples to apples than this.
I think the commenter means the Flash vs Terra benchmarks.
replyAh, my bad. Yeah that makes sense. They do say "The official V4-Flash natively supports the Responses API format and is specifically adapted for Codex.", so at some point someone will make a "same harness" comparison.
replyIt will be fair if they release the harness though. I think now the future will be paired model-harness releases, not just weight dumps.
replyThe performance changes are so big with the right harness that is makes sense to engineer the harness and fine-tune the model to one another from the start.