Yeah giving a 1 hour limit for code generation makes the whole thing a bit incoherent. Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them.
replyIf Astra is leading the pack...Considering it's supposed to be one of the slowest model.
reply> Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them.
replyIf that's the case then these models are failing extremely hard. 7% win rate against human written AIs that they should be copying from memory? A copy paste command should do 50%!
> I am just not sure what it measures.
replyAll of the above, to the extent that they can be measured from a simple to verify end condition (stats, win rates, etc). Adaptability, ability to understand and progress on a task, and so on.
Yes, but I guess my objection is that if it attempts to measure everything then we can't really tell what it measures in what proportions.
replyYep, that’s a feature of end to end measures, like company revenues, vehicle fuel efficiency, or human life expectancy.
replyThe customer may not care what the internal contributions are to such a measure. Instead it represents the customers’ overall values in some way.
Ok. I think you convinced me based on those. I withdraw the objection.
replyAlso: given that the human written bots are almost certainly in the training data, the winning rate against human-written bot seems to be not well-calibrated. You would expect at least they are 50% or better, given that they can "memorize" the best human-written bot.
replyWorth digging into the protocol and see what went wrong.
Top bots are in the ballpark of 50kloc. Even large LLMs are not so large that they can memorize all the world's data like that. And all the fine-tuning that compression loses comprises most of the work in the first place.
replygeneral "agenticness"
reply