upvote
It's pretty safe to say that AI will be used on the battlefield making real life and death decisions before it will be able to render a decent pelican on a bike in SVG.
reply
It already has been and this has been widely written about. AI was used to identify and prioritize targets for the US to bomb in Iran.

Here's an article from 2 months ago for example: https://www.theguardian.com/technology/commentisfree/2026/ma...

It was also implicated in the bombing of a girls elementary school which left 168 dead. The US did a "triple tap" to kill any first responders.

https://www.theguardian.com/news/2026/mar/26/ai-got-the-blam...

https://www.theguardian.com/technology/2026/apr/01/dont-blam...

reply
I read the article and it doesn’t say it was used for targeting or prioritizing?

> Neither Claude nor any other LLMs detects targets, processes radar, fuses sensor data or pairs weapons to targets. LLMs are late additions to Palantir’s ecosystem. In late 2024, years after the core system was operational, Palantir added an LLM layer – this is where Claude sits – that lets analysts search and summarise intelligence reports in plain English

There’s a lot of humans in that loop who make those decisions.

reply
Yeah militaries don't use commercial chatbots for that, they have their own machine learning implementations. Look into Project Maven for example.

And while there are still humans in the loop, the impression I get is that this is increasingly becoming meaningless, from the way they talk about optimizing the "kill chain" and letting small teams make hundreds of targeting decisions per hour.

reply
“US Military Using Claude to Select Targets in Iran Strikes”

https://futurism.com/artificial-intelligence/claude-anthropi...

reply
First link says

> AI is ‘identifying and prioritising targets, recommending weaponry and evaluating legal grounds for a strike’.

reply
It doesn't specify which "AI" though.

These days that pretty much means "somebody used a computer".

reply
“US Military Using Claude to Select Targets in Iran Strikes”

https://futurism.com/artificial-intelligence/claude-anthropi...

It cites the WSJ but that article is paywalled so I shared this one

reply
This later story suggested it was Palantir's Maven, not Anthropic's Claude: https://www.theguardian.com/news/2026/mar/26/ai-got-the-blam...
reply
I think it's beyond decent. I don't understand how people are not more impressed by this. Just a few years ago the only expectation would be garbled nonsense.
reply
Haha, yeah. I tried for it to create a SVG with scissors and it was hopelessly overwhelmed. I think at least the SVG design niche will be safe a little while longer
reply
the battlefield sounds much easier. worst case scenario you kill somebody, but that's what you're trying to do anyways.

if you kill somebody while trying to render a pelican on a bicycle it's a real problem.

reply
In many battlefield scenarios, there is more than one "somebody" on it. The "somebody" that you kill might not be the "somebody" that you intended to kill.

Depending on the how pelicans are created, it is entirely possible to indirectly kill "somebody" due to the externalised costs of global warming etc.

reply
"shift left" on the battlefield. break down those silos. if you have to ask for permission it's already too late. remember the goal. find the bottlenecks in your system and remove them.
reply
I think that's a fair tradeoff. There's no way I'm going back to writing code by hand again. No one deserves that.
reply
Heh? How long were you writing "code by hand" before?
reply
Years and years. It was horrible. No number of misidentified targets will make me go back.
reply
It doesn't sound like you're in the industry you want to be in.
reply
Maybe all along what mattered most to them was making good software that people love, not the day to day part of writing code. Now it’s the industry they’ve always wanted, and less the industry of people who wanted to get paid to write code.

Software engineers who never cared about the higher level product design aspect are finding themselves in the wrong industry. It’s dismal.

reply
> the bicycle frame is the correct shape

No, the handlebar is wrong. The handle bar is rotating the frame instead of rotating the front wheel. The handle bar should be mounted on the same line as the front wheel is.

Hopefully 4.9 will read my comments :)

reply
Could be an extremely high angle stem that just happens to match the downtube angle.
reply
Maybe the pelican is just riding a road bike/gravel bike
reply
I bet someone shares this link every time you post about bicycles, but since I didn't see anyone share it yet in this thread, I'll take the opportunity to do so:

https://www.gianlucagimini.it/portfolio-item/velocipedia/

Turns out even humans can be pretty bad at drawing bicycles :)

reply
As an aside, some of the renders have only a single side connection to the wheel and that is a valid bike design, the Cannondale Lefty front fork only has a left leg:

https://duckduckgo.com/?q=cannondale+lefty&iar=images&t=ffab

reply
On a new model release, you can guarantee two things are in the replies to Simon. One is your link, the other is "surely the models are being trained on this now"
reply
Sure, but no one is trying to force art from most people into about every area in the economy where anyone ever pays for something visual. If you asked professional artists to draw a realistic bicycle, I'm guessing few of them would try to just randomly guess what the mechanical parts looked like
reply
> The most unintelligible drawing has also the most unintelligible handwriting. It was made by a doctor.

Haha

reply
But if you need to draw a bicycle, you wouldn’t pick a random person in the street. You would hire an artist and you’d be guaranteed to have at least a believable one if not a perfect rendering.

No guarantees is why LLM is akin to gambling. Every new context is essentially picking someone out of the crowd.

reply
Glad to see that the "high thinking" level adds a helmet. Always a smart choice.
reply
And yet some people doubt Anthropic's commitment to AI safety
reply
Sadly I think the correlation between this benchmark and performance is starting to break down imo. Still a legendary idea will be remembered and ingrained in the models forever haha
reply
Here's pelicans in all of the thinking levels - low, medium, high, xhigh, max

https://tools.simonwillison.net/markdown-svg-renderer#url=ht...

reply
Max seems to me to be notably better than the others.
reply
low: yolo

medium: redesign bike so peli can reach bars

high: redesign bike so peli can rest on frame

xhigh: yolo

max: big peli reach bars

reply
deleted
reply
I like the way the max pelican has a stern look on his face
reply
Is the output on the max level meant to be missing?
reply
I just fixed that (force refresh). It hit my default 8,000 output token limit, it worked when I bumped that up.

For max I used 25 input, 17,167 output which cost me 43 cents! https://www.llm-prices.com/#it=25&ot=17167&ic=5&oc=25&sel=cl...

reply
You've peed in the pool Simon, this has to be a part of the internal evals by now! You got to try something new - maybe a panda in a canoe?
reply
If these were in the internal evals then the output would be much better. The 4.8 pelicans are pretty meh
reply
Click the link
reply
I really like that thinking level high gave the pelican a helmet.
reply
Hey simonw I love your test, do you think using thinking level "max" makes sense for this test? I would love to see the results about it.
reply
I don't think the API supports "max" as an option, that might just be a Claude Code harness thing.

UPDATE: My mistake, the API does support max. I added a max one at the bottom of this page (cost 43 cents): https://tools.simonwillison.net/markdown-svg-renderer#url=ht...

reply
The legend.
reply
It's funny that we've reached the level where LLMs draw more correct bikes than any random person
reply
Simon, is your pelican test really captures differences among models or should you at least try like 10 times or something to average the random effects
reply
I've been meaning to do a "run 3 times and pick the best" version for quite a while, I should really pull the trigger on that one. Currently it's one-shot only.
reply
You could run 3 times and overlay/average the images to show how consistent they are
reply
Best-of-3 would be cheating, ruin the test, middle of 3 makes more sense
reply
Why would you need the 3rd run if you pick the "one in the middle"?
reply
Middle as in not the best, and not the worst. As opposed to the second generated in sequence.

But not the best/not the worst is somewhat subjective.. so not sure how well that would work.

reply
I think GP meant picking the median pelican
reply
The vast majority (if not all) of these make it impossible to turn, among other fun things. Only out of curiosity, have you tried prompting further with how a bike must operate to see if it does the right thing?
reply
tried it myself, not much of difference

https://gist.github.com/fendy3002/3026a8c4d67d1301666ec40fc0...

looks like the model already trained well on both bicycle and pelicans

reply
I actually like the 4.7 the most, interestingly enough. Not like you can "objectively" weight artistic output like this.
reply
I find the most miraculous thing about 4.7 to be that the pelican is facing left, wonder why the right facing everything is so ubiquitous in these images.
reply
This happened to me in elementary school. We were doing fingerpaintings using plasticine. After all the bikes were hung on the wall, mine was racing the other way... Somehow it really stuck with me.
reply
What do you think it means?
reply
It's facing left but looking right...
reply
Profound political commentary?
reply
[dead]
reply
I don't see how a frame without a headtube can be "the correct shape".
reply
thanks for always providing this very much on time. I'm wondering what the next, harder challenge could be? Maybe some animated svg?
reply
That little red hat on hard mode is sending me. 4.8 has whimsy
reply
Is the "opossum riding an e-scooter" benchmark in the works for Opus 4.8? ;)
reply
For comparison, what's GPT-5.5 producing today?
reply
reply
Lends credence to my vibe-based assertion that GPT-5.5 > Opus 4.7 (and now 4.8), which is why I've cancelled my Claude plan. Opus 4.8 is them seeing it reflected in their own numbers and having to pull stopgap measures to avoid falling behind while they embargo Mythos.
reply
Am I allowed to say that pelican's little helmet is adorable? I can't provide a strong computational proof, or even a shred of anecdata...

...but that pelican's little helmet is adorable.

reply
Eventually the frontier model folks are going to pick up on your pelican on a bike test and bake-in flawless results for that particular request.
reply
4.7 reigns supreme IMO.
reply