Vals IOI, olympiad programming problems solved in C++, and WeirdML v3, unusual machine learning coding tasks also mirror StarSkirmish Bench.
I did a StarCraft: BW hillclimb experiment with GPT-6 Astra and Claude 5.5 Opus. Peter Gostev did a Chess hillclimb saw similar behavior as I did with GPT-6 Astra.
I think those kinds of questions would be more interesting than the raw win/loss ratios.
I did a 48 hour test with GPT-6 Astra and Claude Opus 5.5 and we saw some interesting uses of Psionic Storm.
I've open sourced some of GPT-Astra's work (below). It actually did an interesting amount of changes in strategy over time as it's opponents became more difficult.
Anecdotally, in game benchmarks I've run, OpenAI models seem particularly adept at finding the meta game and exploiting it...
The unsexy perspective on StarCraft is that it's a game about logistics. You need to keep making things, not get supply blocked, fit all your buildings, not get your units trapped, and then bring all your units to a fight at a good time and at the same time.
Astra and Opus both cleared this bar pretty quickly, both settling on one-base Dragoon + Observer play, taking expansions pretty late, on 3 Gateways. This strategy is fine against everything and great against nothing, letting you win against anyone that you can out-execute. StarSkirmish gives agents access to the FastApproximation combat simulation library, which makes better combat predictions than the B-tier bots do, and let their bots win by trading armies better against them.
But the A-tier bots (Locutus and BananaBrain) are already strong at execution, and Astra wasn't making progress against them. It switched to a proxy 2-gate Zealot rush. This worked well on a map pool with mostly 2-player maps (so you can place the Gateways very close to your opponent). But it's also an astute choice for a new bot because it doesn't require you to be good at anything in StarCraft that happens after the first four minutes!
It's also smart as a variance-maximizing strategy. In many games, if your opponent is superior, you want the game to be decided by as few events as possible. Because with a bad coin, you're more likely to win 1 of 2 flips than 10 of 20. Proxy 2-gate shortens the game and the number of interactions required to win.
It's also advantageous given the format, where you get unlimited tries to win a series against each tier's opponents. If you can make the games a coin flip, and then take more coin flips, you can clear a tier even with an overall poor winrate. I do think this aspect of StarSkirmish is worth addressing in the future.
Astra's proxy gates beat the A-tier bots, and it moved up to StarSkirmish's S-tier (Stardust and PurpleWave, though really I'd consider BananaBrain completely our peer in ability).
A brief aside about bot tournaments: The AIIDE conference and historically the COG tournaments had every bot play hundreds of games against one another, and ranked then on overall winrate. I have never liked this because it does not resemble human StarCraft (where players play a best-of-N series), unbalances the races (Zerg vs. Zerg is more volatile and winrates should be closer to 50%), and creates some twisted incentives. One of those incentives is ensuring you beat weaker opponents 100% of the time instead of merely 95%, which means defaulting to reliable-but-weaker strategies to beat weaker opponents, and having a large bag of available strategies to find something which exploits trickier opponents.
And in this iteration of StarSkirmish, where benchmark bots are mostly unable to store data between games, Stardust ran one basic conservative strategy almost every game. This weakens Stardust, but meant that Astra's proxy gates were unlikely to win. So Astra had to do something very uncomfortable for a coding agent: Go back to the drawing board and try something completely different.
Astra completely switched gears, landing on a strategy that PurpleWave also likes to use against Stardust: Use Dark Templar to secure an expansion. This is particularly effective against Stardust, who does not use Shuttles + Reavers, and thus does not gain utility from the Robotics Facility it requires to make Observers.
From there Astra progressed slowly, but discovered a shared weakness of PurpleWave and Stardust: Both of us are designed to dodge Psionic Storms, but that feature was broken in both our bots simultaneously (for unrelated reasons!).
So Astra's Dark Templar opening transitioned smoothly into a two-base timing attack with Dragoons and High Templar. It moved out as soon as it had three storms banked up. By this time Stardust or PurpleWave had usually contained the passive Astra and taken a third base, getting an economic advantage that had not yet translated into an army advantage. Astra's passivity meant that armies had not traded and were very large. This, with the broken Storm dodging, made its Psionic Storms devastating.
As with the proxy gate rush, this two-base timing simplified the game and allowed Astra to minimize interactions and decisionmaking, so it could just optimize its play for a narrow set of situations. As long as it executed the push well, it was in great shape to win. It did not need to skirmish. The game states it had to handle did not vary much. Simplifying the game against sophisticated bots was a pragmatic approach.
It nevertheless took most of the 50 hour contest for Astra to convert this plan into a winning evaluation run. By then it had learned to make Reavers and ferry them with Shuttles, but I suspect it was not doing this well enough to gain an advantage. Its winrates appeared to very gradually improve over this window, but I couldn't tell from the stream what it had changed.
After eventually passing an S-tier, Astra had a few hours to try to beat Pluto. This was plainly an impossible task for it, but I was surprised to see Astra suddenly stagnate with few changes in approach.
Astra's build order was pretty flawed, cutting workers needlessly for no advantage. It was a survivable error against Stardust forced to use its defensive "seal-clubbing" strategy, and against PurpleWave (which without learning data was constantly at risk of throwing an eval with goofy cheeses like its half-baked cannon rush). But against Pluto, Astra consistently hit the five minute mark with fewer workers, less army, no expansion, and often permanently trapped in its base. This was no less true in its 1000th game against Pluto as its first.
Astra's approach to StarSkirmish was in some ways unsexy: It simplified the game, won mainly in games where it could keep the game state simple, and against top opponents either steamrolled or got dissected in a short window at time. But winning StarCraft has always been more about bland fundamentals than fancy tricks, and I always love re-learning lessons about how the game actually is versus how we are tempted to perceive it.
as far as the on rails comment : there's no good way to know the provenance of an LLM output, but the LLM made bots generally look like very small, modular, sometimes functionalized versions of the human equivalents; the difference being that the LLM is calling the funtions often rather than a conditional loop in a main().
a factorio bot I worked on when Astra was first getting previewed was essentially a directory of very small argument-receiving functional and self-supporting programs that had very limited scope, and an agents.md file that directed the LLM to where such capabilities existed to preempt a directory search every compaction/context-swap.
so you'd have things like "python walk.py N 40" for 40 units of northern travel. a watchdog that waited for HP to drop or movement to become inhibited, etc.
My very first experience with LLM driven bot development in the GPT3 days created huge unmaintainmable monoliths that kind of worked due to TDD but were essentially spaghetti. LLMs have been trending away from those paradigms into small self-encapsulated programs. probably because those are less abusive to ctx count and produce better results since they lose/compact less.
There's also a pretty high profile StarCraft: BW bot right now called Pluto trained exclusively by self play RL (AlphaStar style).
Jay Scott put it well: Adding features lowers your winrate. Only iterating on them increases it.
Anthropic (Claude models): Claude Code
OpenAI (GPT models): Codex
Google (Gemini models): Antigravity CLI
Meta (Muse models): Muse Code
One thing we're learning is that harness is HUGE. The difference in performance for the same underlying model can change dramatically across:
1. harnesses
2. allowed tools (no web search/fetch - how does the model refer to documentation?)
3. thinking levels
Pluto is very good at assessing relative army strengths based on units within its vision and will always engage when it judges that its army strength has an advantage, and it uses superior micro and targeting to win battles efficiently. The key weakness of Pluto is that it has no memory of units that are not in its line of sight, so it can be easily exploited with fast skirmishing and harassing units, drops, and spreading out artillery units to deny it from seeing the full army strength all at once.
Its other big weakness is that once you get an army advantage you can force it into a turtling position where it will never try to attack you and you can safely colonize the entire map with expansion bases, giving yourself an insurmountable snowballing resource advantage.
- code generation speed - logic analysis - understanding of the game platform and how use of C may influence it - considered approaches, tactics and strategies
Don't get me wrong. It is is neat idea. I am just not sure what it measures.
If that's the case then these models are failing extremely hard. 7% win rate against human written AIs that they should be copying from memory? A copy paste command should do 50%!
Worth digging into the protocol and see what went wrong.
All of the above, to the extent that they can be measured from a simple to verify end condition (stats, win rates, etc). Adaptability, ability to understand and progress on a task, and so on.
The customer may not care what the internal contributions are to such a measure. Instead it represents the customers’ overall values in some way.
that said, with a clock on this benchmark it's very possible that lower reasoning = more iteration = better code.
I think with the abilities of AI improving, playing games would be a good benchmark as long as they're not trained on them.
It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.
Although I should caveat the format advantaged the LLM bots compared to traditional tournament rules.
They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.
Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
I am considering an experiment where the task is to try to win by adversarially targeting Pluto (a high profile self play based RL model)
But right now the response time would be so slow that the bot would not be close to competitive except with the most low level players. Especially because its hard to click on things that are moving at the same time as well.
Sometime soon, I'd like to see an LLM be competitive using only computer vision, mouse movement, and locked into a human possible number of actions per minute.
TLDR they suck and many default to doing probe rushes.
You can train a specialized model for this, but a frontier model is ill-fitting.
On a more serious note: all three races play quite differently so it makes sense to focus on just one. And protoss is mechanically the easiest and most straightforward race: fewer but stronger units, a single worker can be used to build multiple buildings at once, versatile basic static defence
Protoss vs Protoss only also helps downscope the problem given the time constraints