If you mean for Anthropic in particular, I don't think so. But it's not the first time a major AI lab publishes an incremental update of a model that is worse at some benchmarks. I remember that a particular update of Gemini 2.5 Pro improved results in LiveCodeBench but scored lower overall in most benchmarks.
replyProbably deprioritizing other areas to focus on swe capabilities since I reckon most of their revenue is from enterprise coding usage.
replyIt's frankly becoming difficult for me to imagine what the next level of coding excellence looks like though.
replyBy which I mean, I don't find these latest models really have huge cognitive gaps. There's few problems I throw at them that they can't solve.
And it feels to me like the gap now isn't model performance, it's the agenetic harnesses they're running in.
Ask it to create an iOS app which natively runs Gemma via Litert-lm.
replyIt’s incredibly trivial to find stuff outside their capabilities. In fact most stuff I want AI to do it just can’t, and the stuff it can isn’t interesting to me.
Constantly. Minor revisions can easily "wobble" on benchmarks that the training didn't explicitly push them for.
replyWhether it's genuine loss of capability or just measurement noise is typically unclear.
looking at the system card for opus 4.7 the MCRC benchmark used for long context tasks dropped significantly from 78% to 32%
replyI wonder what caused such a large regression in this benchmark