I do apologize in advance for putting you in spot like this, but there are two things I have been quite curious about.
1. Since 60% of the benchmarks contributing to AA are now private only, how do you guys ensure transparency? More precisely, how can someone verify the transparency in scoring?
2. There are also disclosures about custom benchmarking services offered to AI labs, what does AA do to keep the custom benchmarking services not getting access to the same/similar data that is used in the private benchmarks?
I looked into this using data I have from a battle system built on top of OpenCode (so we're trying to capture real world use). It looks like Carbon actually does a lot better when it uses significantly more output than opponents.
Win rate when output is less than 2.5x opponents: 53% (41/78)
Win rate when output is more than 2.5x opponents: 73% (41/56)
The p-val on this is 0.02. Data from https://agentduel.com/leaderboard.
There doesn't seem to be any relationship between actual size of task though. Win rate appears random based on total output of Carbon and also opponents. (Though our data is still small, so it might exist.)
Looks basically like Carbon was trained to think a lot longer, not necessarily that it's better on large tasks.
I’m out of my depth here but it does seem prompts have a “smell” to their complexity that means we can use a classifier to mate them to a model with better success than random noise, but still don’t essentially all of the savings associated with systems like this just come because there is some “complexity arbitrage” where there is enough of a token cost differential between models that you always save money by trying the cheap model first and then trying a hard model if it turns out to be hard?
What I want is a deeply introspective model that knows to take a step back and call in for aid.
It's really frustrating to describe. Easiest is you can't score on just the prompt, you need the full context. Ex. "Solve Fermat's last theorem" initial prompt then "continue" followup prompt. Which gets you back in the tarpit of "if I have to pay a model to do an inference run on context, then, the context is cached and I should continue using that model."
The vision of another model thats RL maxx'd on knowing how to delegate has a few problems. One, we can observe externally, it died/didn't work even at OpenAI's scale and willingness to make it work, i.e. the...GPT-5.0? IIRC...router that silently went away. The other, if such a model was built, it'd also be good at regulating it's own thinking enough that it'd naturally use less tokens.
In practice, the frontier hackers have settled on a decision model (i.e. Jev) x calibrating effort level or choosing from a set of models x reasoning effort tuples they specify rules for choosing in their Jev prompt. My guess is this stays stable and the only real choice.