They don't have anything competitive with Opus 5.5 yet. There is a supposed model in the works, "Bel" last time I paid attention to gossip, that is thought to be better, later this year. Allegedly Anthropic caught OpenAI off guard and / or their different training approaches ended up working better than anticipated.
replyMy expectation is Fable 5.5 is next overall improvement, and my bet is OpenAI's "Bel" will be the next competitive one (or, their lack of safety culture will bite them/us in the process).
There’s also a chance they surrender the dev market altogether to Ant and just focus on the “normie” segment with dots, computer use, excel etc. In which case they won’t have anything competitive to opus-5.5 anytime soon for SWE.
replyLuna and Sol-6.1 are crazy cost effective and all flows here have switched to them, so I'd say really well so far.
replyThere's very few stuff that needs Astra/Opus level intelligence.
Luna 6.1 would be amazing.
This release is interesting for enterprise at API rates.
replyAt least unless the Terminal Bench 4.0 score is more indicative than the other ones, because that one shows it clearly worse than 6.1 Sol for both performance and cost.
Taking the benchmarks in the article at face value, this is ~on par with Opus 5.5 for ~1/4 the price!
replyWhich is why I don't take the benchmarks at face value...
replySo you have decided the model is bad without having used it yet?
replyIf they had a model on par with Opus 5.5 there would be no need to undercut it by that much, that's probably the logic.
replyAnthropic models are comically overpriced. They also gobble up an obscene # of tokens to get the same result as others. The price delta is literally 5x.
replyNo, I have decided I don't trust benchmarks without using the model first.
replyTaking the benchmarks (as reported by OpenAI) at face value, why wouldn't they show Opus 5.5 on the OS world benchmark, and show no Anthropic models at all on the "factual error rate" benchmark
reply...no it's not? At max effort, it scores 50% on terminal bench vs. Opus 5.5's 70%.
replyALE (top benchmark in post) is 60% for Terra vs 63% for Opus - I guess we'll have to wait for AA Index!
reply48 for GPT-6.1 Terra Max vs 58 for Opus 5.5 Max on AA Index
replyI hate Opus 5.5, constant overreach and over-engineering for any project
replyThat's wild to say when comparing to the OpenAI family of models. I've never seen so many tests that validate tests that validate tests.
replyRight?! I think sol has some obsessive tendencies, i literally caught it the other day spawning a sub sub agent doing a final fourth test for a piece of code that isn’t even critical say in embedded or similar, now I have to instruct it to only test once!
replyMedium or High? Depending on the task, you might want to switch between them.
replyYou’re holding it wrong.
reply