upvote
Given that both Gemini 3.5 Flash (high) and Gemini 3 Flash Preview (medium) beat GPT-5.6 Sol (high) for correctness and score in your benchmarks I don’t trust them at all. The rest of the ranking also doesn’t make sense, like GPT-5.3-Codex (medium) performs better than Claude Opus 4.8 (medium) yeah sure
reply
It's because the benchmark is not coding-only.

Gemini models tend to have most knowledge for most domains, and are one of the most intelligent overall. You can check other benchmarks too, on specific categories, those models still beat other SOTA models.

Regarding Opus, Anthropic models often fail to follow instructions, formatting requirements or simply refuse to answer questions (i.e. Fable).

The issue with Gemini models is that they are not as good as using tools or go into weird failure modes when coding or trying to extract/generate specific data. They work amazing, until they don't...

reply
100% agree with your statement that gemini models have most knowledge in domains. They shine in so many generic topics
reply
One example where the order seems correct, is this SVG generation test:

https://aibenchy.com/showcase/?q=Gemini+3.5%2Cgpt+5.6%2C+5.3...

You can see that most Gemini 3.5 generations are more correct than 5.6 Sol (the net is in the middle of the table, hamster seems reasonable and not deformed, etc.)

reply
It doesn't seem correct at all though? the supposedly best one isn't even the best gemini flash output (the medium one looks better than the high one)
reply
Interesting that Sol (low) did better than Sol (medium) in your benchmark (and is barely more expensive than Terra). I too have been using 5.3 codex as a cheap-but-good model and are switching to Terra (xhigh).
reply
Yeah, for some reason the (low) versions do really well, like they think directly of the solution instead of going around all the edge-cases and getting lost in one of them.
reply
Also for most, there doesn't seem to be a big difference between (medium) and (high).
reply
Here's all 3 (medium), and GPT-5.5

It GPT-5.6 doesn't seem to be a lot smarter than 5.5, but it is faster, cheaper, more efficient and more consistent:

https://aibenchy.com/compare/openai-gpt-5-6-sol-medium/opena...

reply
[flagged]
reply
Where's your website?
reply