upvote
> 98% of the coding quality

Can someone help me understand how they arrive at this figure?

reply
98 can be bad. E.g. uptime or aviation safety.

98 can be good I.e. human test score or funds recovered from debt.

I think for AI tasks it is somewhere between but 2% losses will add up (or you need more loops to correct and hence cost)

reply
Dont worry - it only hallucinates security breaking bugs in 2% of the code it produces. Comparatively :-D

In all seriousness I have no idea how they came to that conclusion. We really need better standards here for evaluating these things but obviously that is difficult.

reply
I think they mean "98% of the average score on these old benchmarks which are no longer representative".
reply
5% of the price compared to Opus 5.5 Max reasoning, and that's at a 50% promotional discount for MAI Code 2, I believe.

With Opus 5.5, Max would be around 4.5x more expensive than Medium reasoning.

That MAI Code 2 is within 4% of the performance at Max seems unlikely, to say the least.

Actual performance and a price comparison to Opus 5.5 at Medium (Anthropic's recommended setting) would be interesting.

reply
that's the thing with coding quality. as we all know, one line of code can ruin everything, so every small increment matters much more than the cost.
reply
How much % of price to reach 200% of performance?
reply
>98% of the coding quality of GPT-6.1

Meanwhile, Deepseek 4.1 flash is 3000% better at 3.5% of the cost, in that it gets work done (I have never had anything really done with GPT 6.1 Sol).

This is real token usage in a complicated monorepo codebase for adding a feature:

---

315,207,850 tok

Cache hit

99.6%

Uncached input

1,187,911 tok

Cached input

313,640,960 tok

Output

378,979 tok

---

It ended up creating a few PRs, with a total of around 6000 lines of code and the feature works as expected. Total cost was around $1.35. GPT 6.1 Sol would have cost around $40 for this and from my experience with it, it would most definitely have had so much bugs that it would not be worth fixing.

So, Microsoft would charge you $16 to waste your time even more. What are we even doing?

reply