I had intended to caveat that: I'm sure I'm not the first person to ask about this!
reply> you still see improvements
This is expected if they are training their models on it, right?
> objectively-bad results
Keen to learn when this has been the case, i.e. across version increments in major models.
I've written about this a couple of times, most notably here: https://simonwillison.net/2025/Nov/13/training-for-pelicans-...
replyI've been enjoying seeing how the quality of individual models differ based on the amount of reasoning effort you give them. If they were baking an a good pelican you wouldn't expect them to differ so much.
(Google Gemini are the only lab that have very clearly paid attention to the quality of SVG animals-riding-vehicles, see their announcement for Gemini 3.1: https://twitter.com/JeffDean/status/2024525132266688757 )
Amazing, thank you Simon! Look forward to reading.
replyHence it has become a meta-benchmark of relative progress in SVG image generation of a known target which has leaked into the training data and for which "every frontier AI team has/had a person at least partially dedicated to" at least checking if not optimizing.
replyI honestly assumed their comment was tongue in cheek humour, because positively no one actually cares how these models generate an SVG pelican riding a bicycle. It's some meme thing that this stuff always appears here.
replyYeah this is not a real benchmark, it's just a fun tradition everytime a new model is released
reply"fun" / boringly predictable meme thread with 30+ replies already
replyIt is telling that people need to create throwaway accounts to criticize simonw's behavior in this website.
replydeleted
reply