undefined

points

by walrus014 hours ago |

comments

by penagwin4 hours ago|

[-]

> [Space Bunny] was an early version of MiniMax M3.1 Flash.

They mention that space bunny was an earlier checkpoints, so it may have improved (or may it could be benchmaxxed)

I find it interesting that so many models seem to be at almost exactly 74% specifically which makes me suspicious of the benchmark. I want to say 5.6-sol, Astra, deepseek flash 4.1, opus 5 (and 5.5?), Gemini 3.8-flash and a few others all ended up there?

by jakswa2 hours ago|

parent|

[-]

> so many models seem to be at almost exactly 74%

There was talk of deepswe tests being flawed. I have yet to find a better gauge of useful work for my case though, so I still keep referencing it and sampling it. On the lookout for improved benchmarks.

"We found issues in at least 23 out of 113 tasks (over 20.3%), leading us to designate the benchmark as Flawed." - https://epoch.ai/benchmarks/deepswe/review

by penagwin2 hours ago|

parent|

[-]

I do agree that it’s generally matched my impressions of models for general SWE at least until we had Opus 5 and 5.6-Sol beating Fable 5. Which coincidentally is when they hit 74%.

Models like Deepseek 4.1 flash, Muse 1.3, and MiniMax 3.1 flash are certainly capable and very cost effective, but I’m not sure anyone would say they’re better than Fable 5/5.1 in capability