upvote
> - Why are they stacking Fable xHigh against Carbon Med?

Because on that specific benchmark (FrontierCode 2.0), Medium is Carbon's best effort level and xhigh is Fable 5.2's best effort level. You can see that on the graph your next point is about.

And it's just that one benchmark that's at medium. "All scores show each model’s highest-available effort unless otherwise noted".

> - How did they produce such a bizarre effort-intel curve for Carbon that goes against every available benchmark?

The graph is labeled as being about FrontierCode 2.0. Those appear to be Carbon's scores on the official FrontierCode 2.0 leaderboard: https://cognition.com/blog/frontier-code-2.0

> - Why did they leave Carbon-- positioned primarily as a cyber model-- off of the single cyber benchmark comparison?

> - Why do they switch to comparing against Argon (rather than carbon) for robodojo?

Because no numbers are available for Carbon for those benchmarks, and the model isn't publicly available so they couldn't run the benchmarks themselves?

And from your followup:

> where's Astra on that CVE chart

To run Astra on a vulnerability finding eval, you'd need to be part of OpenAI's verified cybersecurity program. Otherwise the results would be tainted by refusals. Do you expect that OpenAI would grant such access in this case?

reply
> Because on that specific benchmark (FrontierCode 2.0), Medium is Carbon's best effort level

I am referring to the chart that cross-compares multiple benchmarks, which Artificial Analysis clearly shows Carbon-high as superior on. Further, AA's FrontierCode chart places Carbon-high as having a higher score on Frontier 2.0 than Carbon-med (https://artificialanalysis.ai/evaluations/frontiercode-2-0?m...)

Where it gets very interesting is when you look at the Cognition "FrontierCode" publish date (2026.10.27), and the revision as of 10.28 in web archive (https://web.archive.org/web/20261028175237/https://cognition...). This revision has a VERY different Carbon effort / intelligence curve where high beats medium.

It is only with the latest revision, as of today, that all of a sudden Carbon's benchmark curve goes cone-shaped. Curious, indeed....

I'd encourage you to take a look at a third-party's take on this and see whether Anthropic's self-serving report looks "on the level" after doing so: https://artificialanalysis.ai/?models=claude-fable-5-2-high%...

reply
> Do you expect that OpenAI would grant such access in this case?I

Do you expect that nobody at Anthropic has such access? Trillion dollar company can’t manage this basic level of opposition research?

reply
Best not to give special access to your biggest direct competitor beyond accessing your product at typical API prices.
reply
Probably not for lack of trying though. The incentives for corporate espionage are strong.
reply
Isn't it against TOS for OpenAI to use their models in the development of competing models? So, any employee who does so would be opening themselves up to civil liability at AI scales. IANAL, obviously, so genuinely curious.
reply
Maybe they contract out the work to other companies in other countries who promise not to access OpenAI and then sign up to do things they couldn't do without accessing OpenAI? Like independent promoter payola.
reply
Carbon is still vaporware for most people. You evaluate what you can get your hands on.

Google is still living in its golden age, where it had no real competition, and could move at its own pace.

Time have changed. Googlers better wake up.

reply
Question, why does Google /have/ to compete model for model? If you zoom out on a time scale even slightly the difference in the models is almost non-existent. Google owns the railway and many rails themselves...
reply
People prefer a chat interface to a search box.

OpenAI is coming for google’s cash cow ads business (which is already vulnerable because of their sharp practice).

reply
OpenAI says they're seeing 20% of Google's search volume. [1]

This is a huge deal. The last time any search engine competed with Google it was Yahoo Search, which maxed out at about 40% of Google's search volume around 2006.

Ever since, nobody has even hit double digits. Bing tried and failed, despite getting all of Yahoo's traffic (which was in the single digits by then).

[1] https://www.forbes.com/sites/gabrielalinzainescu/2026/11/05/...

reply
Google isn’t interested in the same way OpenAI/Anthropic are, Google has other sources of like, actual profit, to factor in, eg to resource considerations.
reply
Maybe it's not generally available so they can't benchmark it? Don't even know who has access now.
reply
Because Carbon is not out, nobody can run tests on it.
reply
Arena.ai released carbon tests 12 days ago - the only service that I have seen on yt that gained access. And you can compare to other SOTA models in their video.

https://www.youtube.com/watch?v=NO2I3ueVQrk

And they have one on fable 5.2 that was just released, yay

https://www.youtube.com/watch?v=QwaU4yU0PdQ

reply
Just to follow-up-- artificial Analysis shows no justification for that Carbon effort curve, but it does show Carbon on medium smashing Fable on things like Terminal-Bench and AA-Omniscience v1.1 Non-Hallucination Rate.

But I suppose the reality is rather obvious when you note the funny games they're playing with the graphs, and the things they won't talk about (where's Astra on that CVE chart?)

reply
yes. this.
reply