Do you ever randomize on a particular model - like try and get 3 outputs and pick one at random?
replyCoz who knows if astra low will produce max like output if tried once more.
Btw, they had 3D pelican-on-the-bicycle easter egg in one of the promo videos: https://youtu.be/bOC3DisEOfg?t=117 so I'm pretty sure that they spent some small amount of resources to train the model to produce good svg version as well. :D
replyIt seems to start understanding how bicycles work. Look at the front fork. There are reasons why it's shaped the way it is in real bicycles. If you do physical simulations in 3D with reinforcement learning you will start understanding that most of the arrangements that the other agents use just won't work.
replyYes, the curve of the front fork on the max pelican was what I noticed also. It's impressive (if not an accident), and a remarkably accurate bike overall. The pelican just needs to raise their seat a little.
replyYou would not expect the developers of the model to optimize for a well known benchmark?
replyHere's 'Generate an SVG of a ring-tailed lemur riding an electric scooter' at reasoning level max: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
replyQuote from the thinking trace:
> I’m thinking about how a helmet would obscure lemur ears, but using an electric scooter helmet seems responsible.
It's pretty solid - face is a little wonky but excellent tail and scooter.
Hmm, the face makes it not work as a one shot artifact.
replyIf you want to see something brutal, ask for a zebra riding a scooter. I have yet to see any models do a credible job at that.
replyI would expect it happens more organically where people’s discussion of the benchmark and posted results find their way into the training data, like anything else on the internet.
replyWhat is this supposed to mean exactly? Do you think there are developers at OpenAI or Anthropic whose job is to train these state of the art models to draw pelicans riding bicycles? Like how exactly do you expect them to be doing that anyways? Hiring graphic artists to create SVGs of bike riding pelicans and feeding thousands of them into the model's training set?
replyGenerate two images, get a vision model to judge, then RL reward the better one. And yes, OpenAI employees on twitter bragged about the model's SVG capabilities. So there are clearly people working there who care about it.
replyYou gotta look up how RLHF works before you ask a demanding question like this.
replyThere was a HN post that tested this hypothesis a few weeks ago: https://news.ycombinator.com/item?id=49010129
replyIt stopped being a valuable benchmark proxy quite a few model versions ago. Simon knows it, so is everyone who’s serious about it
replyTreat it like a bit as is
It's actually not just valuable, but infinitely valuable. Because it's imaginary, there is no way to do it perfectly, and thus the possibilities are endless, and it tests out both the formulation and execution of ideas.
replySimon do you have a page somewhere that shows _all_ of the penguins created by various models across time?
replyYou’ve been doing this public service for so long (well, for so long in “AI hype” years anyway) that it’d be fascinating to see the evolution of this artifact across time.
For the moment the best place to see them all is to browse the tag on my blog - 140 posts now! https://simonwillison.net/tags/pelican-riding-a-bicycle/
replyI'm running out of excuses not to build a proper comparison site though. Maybe I'll have Astra do rhat.
Would be great to spend some money (other people's money) on paying some illustrators and artists to do the same for comparison. It could be straight images converted to SVG or direct vector graphics.
replyI find curios that Astra's pelicans are basically the same (yellow sun top right corner, green bike, same bike shape, same legs style, very similar background) while in other there is more randomness.
replyThis tracks with what OpenAI has been saying about Astra: that it tends to do things in a certain way and it’s up to you to prompt it to change its style.
replyAstra is the only model that correctly depicts occlusion of crank and leg, although interestingly, at max and medium levels (not in between).
replyLooked to me like it missed the chain though.
replyAstra pelican looks amazing: it looks like it was made by a professional artist. All the others look like they were made by kindergartners.
replyInteresting that all of the bikes are turquoise colored, except for the medium effort
replyOh my goodness. I was not prepared for Luna on "none".
replyReminded me of https://clocks.brianmoore.com/
Haiku the GOAT
replydeleted
replydeleted
replyDo you have other ideas of "combinations" for this kind of benchmark?
replyI'm wondering if this is being trained on by the models today.
I have a feeling they've been doing that for a while now, even if unintentional, as it's such a well known benchmark
replyCrazy how much the Astra pelicans resemble GLM 5.3's (https://crimson-jeri-74.tiiny.site/), down to the color of the bike and the scarf.
replyThe improved front fork design mentioned by Threatripper is about the only thing Astra is doing better, IMO.
radial spokes on the back wheel is not a thing irl. i wonder if any model has gotten that right?
replyYou’re closer to this than I am but do you find it odd that helmets are almost never included? Biking is almost always accompanied by helmets
replyBut pelicans are almost never accompanied by helmets, so it cancels out.
reply> Biking is almost always accompanied by helmets
replyIsnt Netherlands the leader in bike riders and they dont wear helmets.
At least one reasoning trace I saw considered a helmet and discarded the idea because it was worried about obscuring some detail
replyLuna's price seems off
replyThe price is probably coming from when Luna was first released. OpenAI slashed the price by 80% at the end of July.
replyYes! Good catch, thanks - I'll fix that.
replyI had the prices wrong on Sol and Terra as well - they've all had price drops:
replyhttps://openai.com/index/advancing-the-price-performance-fro...
Sol discount is until November 21, 2026 according to https://developers.openai.com/api/docs/changelog
Luna — costs in cents
+--------+--------+---------+
| Effort | Before | After |
+--------+--------+---------+
| max | 7.83 | 1.57 |
| xhigh | 4.24 | 0.85 |
| high | 2.46 | 0.49 |
| medium | 1.26 | 0.25 |
| low | 0.76 | 0.15 |
| none | 0.71 | 0.14 |
+--------+--------+---------+
Per million tokens:
Before: $1 input / $6 output
After: $0.20 input / $1.20 output
Sol — costs in cents
+--------+--------+---------+
| Effort | Before | After |
+--------+--------+---------+
| max | 48.55 | 32.37 |
| xhigh | 24.11 | 16.08 |
| high | 10.38 | 6.92 |
| medium | 10.55 | 7.03 |
| low | 8.33 | 5.55 |
| none | 5.90 | 3.93 |
+--------+--------+---------+
Per million tokens:
Before: $5 input / $30 output
After: $4 input / $20 output
Terra — costs in cents
+--------+--------+---------+
| Effort | Before | After |
+--------+--------+---------+
| max | 32.09 | 25.67 |
| xhigh | 14.67 | 11.74 |
| high | 3.74 | 2.99 |
| medium | 3.46 | 2.77 |
| low | 3.47 | 2.78 |
| none | 2.60 | 2.08 |
+--------+--------+---------+
Per million tokens:
Before: $2.50 input / $15 output
After: $2 input / $12 outputFallen off the homepage so what? I genuinely don't understand
replySo he is posting it again because he believes it is crucial information for us.
replyApparently the crowd agrees because they keep upvoting these.
Luna is my go to daily model it's great value
replyMe toooo, for all my personal projects I need to pay for the tokens~
At workplace I use sol because I don't need to pay
replyMine as well, it’s fast and keeps me in flow; and cheap!
reply[dead]
reply[dead]
reply[dead]
reply[flagged]
replyHow are we supposed to know if Astra is frontier without the pelican?
replyIt's simonw. It's interesting to see their pelican benchmark, another comment by a different author elsewhere here shows some very good SVG generation too.
replyIt's absolutely related.
reply[flagged]
reply