Oh, it coded a Pac-Man clone. The clone was so good that I thought it was premade in some way and that Sonnet was going to play PacMan.

I have played few of them and it seems that Opus 5.5 is the first one who really made playable PacMan clone game. On mobile as well.

Could it be because the model was somehow pre-trained? If we compare it with pelicans that are still not-perfect…

Cool page and benchmark idea! Would be nice if there was some kind of grading the results, maybe on different criteria (aesthetic, implementation complexity, correctness, ...). Of course as a one-shot and greenfield benchmark the results are not indicative for all kinds of usage patterns. But as some sibling said, maybe they can be indicative on some general characteristics (especially since the task is so open-ended).

Very cool. I'd love to see someone with access to plenty of token$ make something similar for the "Browser Desktop OS" test. That seems like a pretty comprehensive test thats also fun to test just like this!

Thank you for doing this. It is very helpful not just for capabilities but also for costs.

I was able to get similar with Qwen 3.8 27B with one shot. I think this game is too well in the training data.

GPT models really have no taste huh.

Interesting. Sonnet 5 was horrible, and Opus 5 was unplayable, but both Sonnet 5.5 and Opus 5.5 were about as close to the real thing.

Wow, that "bake off" page is better than any coding benchmark I've seen! You can really sense the strengths and weaknesses of each model/harness combo.

oh! How about pengo, dig dug, & defender?

Mainly because they're funny, but it's also because I try pretty hard to make the comment more interesting than just "here's a pelican". In this case I used the pelicans to talk about the 128,000 token limit bug at "max" and share comparative pricing.

In the GPT-6 comment I included full visual comparison grids: https://news.ycombinator.com/item?id=49805509#49806126

For DeepSeek v4.1 Flash I identified that the OpenRouter reasoning levels are mapped to a smaller set of levels for that model: https://news.ycombinator.com/item?id=49639090#49645591

Because hn has some kind of a community and not every comment is gold (see yours for example) and people are able to skip comments if they don't enjoy them?

It's an easy way to compare the coding and creative strengths of models. I prefer them over reading a tabular comparison of benchmarks which you have no real insights into.

Agreed. It was a creative and unique test for a while. Now, no offense to the author, it feels like every conversation about a new model is dominated by the pelican on a bike posts as they always become the top comment.

Karma farming by parent commenter and HNers’ tendency to upvote low quality content (not dissimilar to other social media networks)

If it was trained on HN, there would be a 60% chance of it just saying "I'm so tired of this request, can we please move on"

Anthropic's "Max" modes seem like a yolo mode: "use 10x the tokens to try to break the hardest possible problems". But their models don't seem less efficient at normal reasoning modes.

I can't see Ember on AA's index yet, but their post claims "half the reasoning tokens for the same answers" as Kimi K3.

That would make it about so, I assume?

             Score  Tokens  Reason  Cost 
 Kimi K3 Max    44  48k     32k     $2.00
 Half reason    44  32k ?   16k ?    ?

 Opus Med       51  26k     12k     $1.34
 Opus High      54  36k     18k     $1.82
 Opus Max       58  119k    84k     $5.98

 Sonnet Med     41  ?       ?       $0.59
 Sonnet High    47  ?       ?       $1.08
 Sonnet Max     56  193k    142k    $7.60

Medium is Anthropic's default.

Having a less efficient mode isn't necessarily a mistake -- the purpose of configurable effort levels after all is to be able to put more thought into a problem.

That's max output tokens per response limit, separate from context length

It's the output token limit, which has been 128,000 for Claude models for quite a while note

wow nobody but you has ever thought of this and certainly simonw has never addressed this