The SWEBench benchmarks are really gamed at this point and should not be trusted period. The solutions are effectively in the training sets and have been for a while.
replySWE Bench Pro is completely different benchmark than SWE Bench (e.g. Verified) suite was. It only copied the name.
replySWE Bench Pro is also gamed and shouldn't be trusted.
replyThen we are left with what? FrontierCode maybe? IIRC that one evaluates not only if tests pass but also code quality - e.g. whether the maintainer would accept the pull request as is.
replyWhy that benchmark in particular?
reply