I recently discovered this same lesson the hard way. I was trying to get a multi-agent system I was building to improve upon GPQA Diamond scores (system here: http://pellmell.ai). No matter how hard I tried, I could not get any lift. When Fable 5 dropped, it also did not improve upon Opus, and I realized my mistake. The benchmark was saturated!
Now, looking at the result here, I see a similar pattern. Fable is not better than Opus, and the score is ~95%. Notably, this post omits which subagent is being used. Why? An intellectually honest way to tell if this thing really works would be to run that agent and report its score and cost as well.
Going back to my GPQA Diamond lesson, you can see here how a saturated leaderboard behaves https://artificialanalysis.ai/evaluations/gpqa-diamond. Fable gets 92.6% for $0.22 per task while several models score higher for $0.01. I could easily publish a router that “enhances Fable on GPQA Diamond” showing improved score for lower cost, just by implementing a router that picks the model at random!
On the saturation point, we agree that a 95.8% result on a mature benchmark isn't the main proof, which is why we're currently running against harder, less saturated benchmarks like Terminal-Bench, CursorBench, and SlopCodeBench (going to publish results on these hard benchmarks shortly). Apart from current user experiences, that will show the value of our harness.
Hell, I could publish better score at 87.5% time reduction by having the router always pick Flash!
Dear YC, please force the startups to dedicate some of the 500k funding for standard web design.
Looks like "Techno music design of the 90s" or similar :-D
I guess however this is a harness and it needs to connect to the API?
Is it even hidden when your AI ends up tagging it with `aria-label="Hidden secret code"`? Lol. Fun mini game.
Cmd+Option+I > Console > 'allow pasting'
const onboarding = document.querySelector('#onboarding'); const app = document.querySelector('#app'); onboarding.style.setProperty('display', 'none', 'important'); app.inert = false; app.removeAttribute('aria-hidden'); document.querySelector('#prompt')?.focus();
Also, warning:
# Share chats with Bullet — helps us improve model routing and answer quality
Enabled by default.
Bulletproof.
"The bullets" are not any different than other new agent projects. New agent projects pop up and die constantly and it's not from a lack of funding.