In light of recent pieces like "The Emperor Pelican Has No Clothes" (https://arcfu.com/post/the-emperor-pelican-has-no-clothes/), I'd be more curious to see how 'PelicanBench' becomes more robust in the time to come -- I feel like all models are now benchmax'd on the standard pelican SVG rendering task and we need something more multi-dimensional. It would be great to have a script that prompts a model to render the pelicans in several different situations (like the linked article) that showcase some kind of broad generalizability or other sort of 'transfer learning' going on with these models that would really then be impressive and worth noting on - otherwise PelicanBench is losing its relevance.
replyIt's interesting the high reasoning pelican is animated. I don't think I've seen another model do that before.
replyA question I have about the pelican benchmark : why are the pelicans always cycling to the right?
replyThe gears etc on bikes are customarily on the right, so almost all photos of bikes (eg for sale, by hobbyists) face to the right to show off that, and so it makes sense for LLMs to have such a preference.
replyThey’re all derivatives, not originals.
replyI really would love a browser plugin that blocks having to read about fucking pelicans on every model release.
replyYou could unironically vibe code this with a prompt shorter than your complaint.
replySo make one. Not everything is for you.
replyI think it is safe to say that Muse models are not pelican-benchmaxxed.
reply