I'm not sure about the answer here, but this can be caused by the scoring rubric used by given benchmarks. For instance, if a benchmark docks scores for running too many commands or using too much wall-clock time, higher efforts will get lower scores.
replydeleted
replyMaybe it's akin to the Ballmer peak: improved performance at a specific level of relaxation
replyI agree, very odd they did not comment on any theories for the degradation here. Dip and then rebound at max effort is pretty interesting too. Overthinking is bad, but you can overthink so much it starts to be better again?
replyIt is probably from randomness. The benchmark tasks nowadays are so long that you can't really afford to run a large number of samples of them per model & effort combination
replyNo it's a mean of 5 runs.
reply> We report FrontierCode’s overall score, a composite measure that grades each patch on blocking functional criteria (held-out unit tests) together with weighted code-quality rubric criteria, as mean@5.
They don't explain more in the system card, I guess higher effort levels could loose points on the code quality / scope / style / maintainability stuff?
apparently it got docked points for editing files out of scope
replyThis must not be weighted very heavily on the benchmark because if it was, Opus would bomb every test (half kidding)
replyDo you have a source for this? That would explain it, but could be a bit of a concern on the general focus the model at higher thinking exhibits.
reply