I think we want to move beyond code based bots asking the LLMs to try to train a custom neural net, in a limited time + compute, to play StarCraft BW is probably the most promising direction
replyI am considering an experiment where the task is to try to win by adversarially targeting Pluto (a high profile self play based RL model)
If you are talking about take a screen shot, analyze move the window a bit, take another screenshot, move the window, repeat until they have seen enough of the map to make a decision, then decide what to click on, issue order.... this could absolutely be done. It's very similar to how LLM's have driven a car on a safe track.
replyBut right now the response time would be so slow that the bot would not be close to competitive except with the most low level players. Especially because its hard to click on things that are moving at the same time as well.
Sometime soon, I'd like to see an LLM be competitive using only computer vision, mouse movement, and locked into a human possible number of actions per minute.
https://broodbench.com/
replyTLDR they suck and many default to doing probe rushes.
Doesn’t this suggest that the results seen in the OP are merely copying strategies it has seen in other bots (code in the training data), and not actually the agent developing its strategy?
replyBasically yes, but the space of "AI for RTS and videogames" likely goes well beyond Starcraft, so there's probably a lot of unique programming techniques these frontier models can apply for writing AI bots that haven't been done yet.
replyEven the most powerful frontier models on SOTA hardware are not fast enough for starcraft, which generally involves multiple informed decisions per second at peak moments.
replyYou can train a specialized model for this, but a frontier model is ill-fitting.