SWE-Bench pro is pretty much useless now even though many ppl still look at it. OpenAI published a report yesterday saying so as well. Only look at DeepSWE and FrontierCode right now for coding imo.
replyAmazing, a company that does poorly in a benchmark says that benchmark is useless...
replySWE-bench series just aren't that great by today's standard, even Anthropic previously stated Claude had memorized solutions for the non Pro version of the benchmark, I suspect the recent increase in the score for the Pro version probably also had similar behaviors.
replyBut anyway, I think it's pretty useless to look at SWE Bench's now when other way better benchmarks exist.
> Anthropic previously stated Claude had memorized solutions for the non Pro version of the benchmark
replyyeah that was the point of introducing the Pro version
You've overstated the conclusion. The SWE-bench series has had issues since its inception.
replyOpenAI no longer recommends SWE-Bench-Pro as a benchmark: https://openai.com/index/separating-signal-from-noise-coding...
That's smart, they only recommend benchmarks that make them look better then their competitors.
replyMakes sense why they released an entire study yesterday discrediting SWE-bench Pro.
replyAnd they'd be right, it's an almost saturated benchmark where even some subpar open source models score very well on. And most models are clustered within a small range so it really doesn't tell you much.
reply[dead]
reply