Happened to be testing a "review patches on a mailing list" harness I was developing; here are a sample of the latest results, testing 12 patches containing a total of 14 issues:
replyOpus 5.5: Found 8/14 issues. Total cost: $15.40
Fable 5.1: Found 7/14 issues. Total cost: $66.34
Opus 5: Found 6/14 issues. Total cost: $15.19
Sonnet 5: Found 2/14 issues. Total cost: $19.15
This is a relatively small sample size, but it was both the best and the cheapest.
ETA: NB this is "Equivalent API" cost as reported by claude's CLI; I was using my subscription.
They're claiming a drop in token use too, and that it nets to 40% cheaper.
replyUnfortunately, they're full of it https://artificialanalysis.ai/models/claude-opus-5-5#token-u...
replyIt does work out to be a similar cost per task though
You should probably look at the cost/score graph by effort level instead:
replyhttps://artificialanalysis.ai/models/claude-opus-5-5#intelli...
It is most of the pareto frontier.
Not disputing the increase in quality, just stating that non-cherry-picked benchmarks show it is more verbose at Max effort
replyso don't use it at max? The benchmarks suggest that high/xhigh are more than sufficient to be ahead and a whole magnitude below max with regards to token usage. I'd treat that as an outlier and not how verbose the model is in general (QED I know)
replyIs verboseness the only measure of token efficiency towards overall task completion?
replyDisagree.
Our internal company tests showed a cost per task drop from 0.35usd to 0.16usd . Opus 5low vs opus 5.5 low
replyVery fast you were.
replyEven created an account to tell us just that.
replyI don't think so, I typically use Opus 5 on High, and 5.5 scores lower on token use:
replyhttps://artificialanalysis.ai/models/claude-opus-5-5?models=...
I tested it with Claude Code, and I can confirm it's way cheaper, better, faster and less verbose than Opus 5.
replyparent means that they could get more client / a larger part of the market, which would lead to more income (more tokens) despite lower marginal prices
reply