I don't think this is relevant on AA's FrontierCode score. AA runs FrontierCode 2.0 benchmark using their own Stirrup agent harness [1].
reply[1] https://artificialanalysis.ai/evaluations/frontiercode-2-0
My mistake, thank you for catching.
replyIt's interesting that the Stirrup scores are somewhat close to the specialized harness scores. Either Stirrup is quite excellent or is it that harness doesn't matter so much after a point?
What's the advantage for cognition?
replyI would think it's the opposite as a more competitive frontier for llms makes the labs shed some margins and enable a broader ecosystem.
Yes, that is true, you might be right. But I think such, because the moat in their wrapper layer feels very thin and does not make sense in a world full of 20+ wrappers around similar models. It feels better to "only" have to deal with Ant/OAI (and other pure wrapper cos). Competing vs. Google, Meta, and every Chinese lab on earth, every one using their own bespoke wrappers, is much scarier.
replyAdditionally: competitive frontier = more commodified = high valuations under more scrutiny.