upvote
It's because the benchmark is not coding-only.

Gemini models tend to have most knowledge for most domains, and are one of the most intelligent overall. You can check other benchmarks too, on specific categories, those models still beat other SOTA models.

Regarding Opus, Anthropic models often fail to follow instructions, formatting requirements or simply refuse to answer questions (i.e. Fable).

The issue with Gemini models is that they are not as good as using tools or go into weird failure modes when coding or trying to extract/generate specific data. They work amazing, until they don't...

reply
100% agree with your statement that gemini models have most knowledge in domains. They shine in so many generic topics
reply
One example where the order seems correct, is this SVG generation test:

https://aibenchy.com/showcase/?q=Gemini+3.5%2Cgpt+5.6%2C+5.3...

You can see that most Gemini 3.5 generations are more correct than 5.6 Sol (the net is in the middle of the table, hamster seems reasonable and not deformed, etc.)

reply
It doesn't seem correct at all though? the supposedly best one isn't even the best gemini flash output (the medium one looks better than the high one)
reply