Was it ever a good test? How do you even objectively assess what a good pelican on a bike is anyway?
replySVG generation is a good test because it's extremely easy to subjectively assess with visual reasoning where humans are strong. However, pelican on a bike specifically may be overused at this point.
replyThe "big beak!" comment in the svg source makes me think it's definitely a gamed "benchmark" at this point.
replyDo you think the models are ready for the next level? I believe that would be: Pelican feeding Spaghetti to Will Smith.
replyVariations of this comment have been posted for over a year. The pelican has now morphed into part of HN culture rather than a legitimate benchmark, but it's still valuable as a meme.
replyit is more an example of gaming (the HN system) than meme.
replyI'd be very surprised if this is in the training data given that most models mess it up to this day. E.g. look at the ones from Opus.
reply[flagged]
replyI really don't understand what's interesting about this test and why is it always on top.
replyIt's funny.
replyIt really is lol
replyAs often happens with random oddball things which become traditions in web communities, the replies asking what it is or complaining about it, begin to gain their own humor value.
replySame reason you would always see the same top comments on reddit during a certain era.
replyThat’s what I think too, but we should actively go against such culture here because hn is not reddit.
replyIt basically is at this point, if you haven’t noticed. Complete with the same America bad, Elon bad, democrats good midwit progressive politics.
replyAlmost all Musk related negative news gets [flagged] and never hits the the front page, so there is still a silent base on the other "team" apparently.
replyDon't forget EU bad! Because they won't let Apple screw over consumers.
replyElon does suck. Objectively.
replyIs this Straw Man and Ad Hominem ?
replyIt has become a funny meme, much like "My hovercraft is full of eels!"
replybecause you can't still ask LLMs to port DOOM to hardware X or Y
replyIt's a meme, and HN loves upvoting memes. Just like Reddit!
replyThe ultimate measure of an LLM is whether it can produce a capable image of a pelican riding a bicycle. All other use cases are but a distraction!
replyDo you seriously have a dedicated “bad takes on AI” hn account?
replyyeah, although I do combine it with "replies to snarky questions" for efficiency
replyTrue that
reply