Oh, it coded a Pac-Man clone. The clone was so good that I thought it was premade in some way and that Sonnet was going to play PacMan.
replyYes! The point being that up until yesterday, every model struggled with this, and now they don't.
reply"this" being recreating Pacman specifically, or games?
replyI made ~10 games with opus 5.5 (all multiplayer web games over web sockets).
replyAbout half the time it made a playable game in a single short prompt. The other half of the time a few follow-up prompts were needed for refinement (eg. Things like "the blaster weapon is way too powerful, divide it's hit points by 10" or "we need a way to reconnect a player whose network dropped mid round" or "the GPS doesn't work on iOS")
Other models fail at oneshot creation of similar games?
replyYour welcome!
replyI have played few of them and it seems that Opus 5.5 is the first one who really made playable PacMan clone game. On mobile as well.
replyCould it be because the model was somehow pre-trained? If we compare it with pelicans that are still not-perfect…
Cool page and benchmark idea! Would be nice if there was some kind of grading the results, maybe on different criteria (aesthetic, implementation complexity, correctness, ...). Of course as a one-shot and greenfield benchmark the results are not indicative for all kinds of usage patterns. But as some sibling said, maybe they can be indicative on some general characteristics (especially since the task is so open-ended).
replyVery cool. I'd love to see someone with access to plenty of token$ make something similar for the "Browser Desktop OS" test. That seems like a pretty comprehensive test thats also fun to test just like this!
replyGPT models really have no taste huh.
replyThank you for doing this. It is very helpful not just for capabilities but also for costs.
replyI was able to get similar with Qwen 3.8 27B with one shot. I think this game is too well in the training data.
replyInteresting. Sonnet 5 was horrible, and Opus 5 was unplayable, but both Sonnet 5.5 and Opus 5.5 were about as close to the real thing.
replyoh! How about pengo, dig dug, & defender?
replyWow, that "bake off" page is better than any coding benchmark I've seen! You can really sense the strengths and weaknesses of each model/harness combo.
replyThanks!
reply