upvote
Hi,

John here from the team that built Microsoft Decision 1 and 1.1.

Microsoft Decision 1.1 is on the Artificial Analysis Decision Index v2.1 leaderboard, where score 68, above Gemini Decision 1, GPT-6 Luna Decisions, and Jev 1.14. [1]

We also compare ourselves to other decision models in our blog post on Microsoft's Command Line blog from November 17 [2]. We name all of the public benchmarks we evaluated against in the post, and provide a breakdown by category.

[1] https://artificialanalysis.ai/evaluations/artificial-analysi... [2] https://commandline.microsoft.com/microsoft-decision-model-g...

reply
this doesn't bring a lot of confidence when you are using old version of Terminal Bench or Frontier Code when new variants were available for quite quite some time.
reply
I could be wrong, but I think Artificial Analysis typically runs benchmarks independently after models become publicly available.
reply
A lot of times they are given early access so they can release benchmarks on the day of the model public release.

But the odd thing is that even the previous generations of these models aren't on Artificial Analysis. It's been months since version 1 of some of these models was announced, but none of these are on AA. Are they only providing them with anti-benchmarking language or something?

reply