> The benchmark is saturated. Frontier models are tested with an armadillo in fishnet tights jaywalking on Mars.
replyOK well I couldn't resist this one:
llm -m claude-opus-5.5 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mars'
llm -m gpt-6.1-sol 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mars'
llm -m gemini-3.8-flash 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mars'
llm -m mistral/mistral-large-4 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mars'
The rover honking is pretty silly, opus has a good sense of humor
replyBig L for mistral in this benchmark. Sorry Europe.
replyNot so sure, apparently it is the only one that considered that there are no paved roads on Mars.
replyTo be fair, they don't have Armadillos in Europe. of course, you could say the same for Mars...
replyI'm getting 403 inside the tool for this one. (The pelican bike on top works) This has been happening a lot recently.
replyThat's a GitHub rate limit. Try again now, I just pushed a hopeful fix: https://github.com/simonw/tools/commit/7793fb74c2d37bd613cdc...
replyIt does seem to, thanks!
replyGemini wins this one clearly. Honey please!
replyhttps://chatgpt.com/s/m_6ac53d4e5b0c8191949050dbf1f402d7
replyNot sure I'd call it jaywalking exactly but pretty good