Which is why I don't take the benchmarks at face value...
replySo you have decided the model is bad without having used it yet?
replyIf they had a model on par with Opus 5.5 there would be no need to undercut it by that much, that's probably the logic.
replyAnthropic models are comically overpriced. They also gobble up an obscene # of tokens to get the same result as others. The price delta is literally 5x.
replyNo, I have decided I don't trust benchmarks without using the model first.
replyTaking the benchmarks (as reported by OpenAI) at face value, why wouldn't they show Opus 5.5 on the OS world benchmark, and show no Anthropic models at all on the "factual error rate" benchmark
reply...no it's not? At max effort, it scores 50% on terminal bench vs. Opus 5.5's 70%.
replyALE (top benchmark in post) is 60% for Terra vs 63% for Opus - I guess we'll have to wait for AA Index!
reply48 for GPT-6.1 Terra Max vs 58 for Opus 5.5 Max on AA Index
reply