Well, it is a silly test, not a scientific benchmark.

However, I would say it is a measure (not the measure). If you look at the entries, there's a lot of variation - definitely not something they memorized outright.

And the test itself is deceptively simple. You need to do canvas rendering, there's pathfinding, command queueing, terrain generation, etc. There are some subtle click handler bugs (various LLMs often stumble on those). And I ask the model to do it all in one file, further increasing the complexity of the task.

And the result is something that you can instantly evaluate. And if the result is any good, even play! So yeah, I think it's a fair test.

I'm sure it'll get saturated at some point. Actually I started with Minesweeper and switched to RTS last December, because Minesweeper was being saturated. I'm expecting (hoping?) the RTS test will last until the end of this year...

Ask it to generate a completely bananas game idea and it will easily do it. Prototyping any type of game with simple graphics has been solved since many generations of models back. Claiming it's because it has that game in its training data is nonsense.

The 5.6 model article for this post has three examples of little web games.

There are plenty of little js web games anyway. The point isn't to make an actual game, it's to show coding ability, design and taste in a way that's more assessable than reading a codebase.

The goalpost velocity is approaching light speed…

>There's probably a better base RTS on github somewhere for free.

I... I think you are missing the point.