> so many models seem to be at almost exactly 74%
replyThere was talk of deepswe tests being flawed. I have yet to find a better gauge of useful work for my case though, so I still keep referencing it and sampling it. On the lookout for improved benchmarks.
"We found issues in at least 23 out of 113 tasks (over 20.3%), leading us to designate the benchmark as Flawed." - https://epoch.ai/benchmarks/deepswe/review
I do agree that it’s generally matched my impressions of models for general SWE at least until we had Opus 5 and 5.6-Sol beating Fable 5. Which coincidentally is when they hit 74%.
replyModels like Deepseek 4.1 flash, Muse 1.3, and MiniMax 3.1 flash are certainly capable and very cost effective, but I’m not sure anyone would say they’re better than Fable 5/5.1 in capability