upvote

points

by jmugan4 hours ago |
comments
upvoteby IsTom2 hours ago|
next
[-]
Aren't they still bad at understanding how bicycle frame works? Especially the steering part?
reply
upvoteby gegtik2 hours ago|
parent|
next
[-]
Maybe that makes them human..

https://www.booooooom.com/2016/05/09/bicycles-built-based-on...

reply
upvoteby kees9936 minutes ago|
parent|
next
[-]
I've fed a couple of those chicken-scratch sketches to nanobanana with prompt "Treat attached as a technical drawing of a bicycle. Produce photo-realistic image of the actual bike manufactured to that spec. Try to stay close to the input, where possible", and... wow, results are not great.
reply
upvoteby mgfist52 minutes ago|
parent|
prev|
next
[-]
That was very fun, thanks for the link
reply
upvoteby runarberg1 hours ago|
parent|
prev|
next
[-]
No. But it reveals something about how this technology works, its statistical properties, from which you can infer the limitations.
reply
upvoteby nozzlegear1 hours ago|
parent|
prev|
[-]
Humans aren't machines trained on the entire stolen corpus of human knowledge. We expect that a human will do poorly at arbitrary tasks they have no experience doing – especially drawing, which many (most?) humans aren't trained in at all. The same isn't true of AI, where its proponents, priests and proselytizers have spent time, energy and billions of dollars attempting to convince us it can do anything better than humans.
reply
upvoteby nozzlegear1 hours ago|
parent|
[-]
Pelican enjoyers didn't like this one lol
reply
upvoteby bfung56 minutes ago|
parent|
prev|
next
[-]
I don’t know what datasets are available to these LLMs, but I’d imagine if there was training on CAD code, text, and images, a prompt steered towards that probably could get it pretty good.

I am not a mechanical engineer, so even prompting well with ME lingo probably will take some effort.

reply
upvoteby zh32 hours ago|
parent|
prev|
next
[-]
Shhh...you'll alert the models :)

Totally agree though, anyone with a vague understanding of how bikes works ignores the pelican because they know the bike is unrideable in the first place.

reply
upvoteby edaemon2 hours ago|
parent|
prev|
[-]
Yes, but I think the idea here is that most models produce very similar pelicans on bicycles, so a different test might be more useful in gauging the differences in models.
reply
upvoteby maxutility4 hours ago|
prev|
next
[-]
Agree. The pelican benchmark was interesting a year ago when most models struggled and a good pelican indicated an unusually capable model. Now it’s saturated and uninteresting.

A good new benchmark should have awful performance to start and there should be a lot of headroom for improvement. This benchmark is also intentionally difficult and requires the LLM to develop the animation through spatial reasoning and first principals rather than existing video generation pipelines. Similar to how SVG generation was out of distribution for most models a year ago.

reply
upvoteby irthomasthomas1 hours ago|
parent|
next
[-]
I think it's interesting to see them visibly struggling to improve. Claude pelicans aren't much better today than they where 18 months.
reply
upvoteby sixtyj1 hours ago|
parent|
prev|
[-]
Have you seen pelicans in Simon Willison’s tests? It is still not a pelican on bike I would like to publish :)

Imho we don’t need to make benchmarks that draw the whole 3D world. Pelican’s drawing is really nice in its simplicity and complexity at the same time.

It seems to be an obscene waste of compute time to generate useless 3D worlds that are just a bragging - 3D is really heavy discipline to make it right, see Mark Zuckerberg’s ceased attempt with 3D VR…

Multiply it by thousands times as a lot of people have found out threejs lib and prompt “generate 3D world and make no mistake” are new orange/black.

reply
upvoteby user4392851 minutes ago|
parent|
[-]
Do you expect the SVG to emulate a hand-drawn picture, become more realistic, or just a more detailed illustration?

As for the often quoted issues with the bike's frame or problem with the steering column, I can't really tell, I am no bike expert.

I can instead judge how poor of a job it is doing with a LOTR rendition in Three.js, so that seems like a better benchmark.

reply
upvoteby sixtyj12 minutes ago|
parent|
[-]
A general benchmark (even Simon mentioned that it was meant as fun at the beginning) should be quick and easy to run, since we can expect that more people will want to try it out. That’s why I’m more like “team Pelican on a Bike”… :) cheers
reply
upvoteby dofm3 hours ago|
prev|
next
[-]
But it's another benchmark on how good models are at generating intensely average, unwanted things with unthinking design. Just scaled up.
reply
upvoteby charcircuit2 hours ago|
prev|
[-]
Bad? It has a charming style. I would watch the whole book if it was made like this.
reply
upvoteby trial32 hours ago|
parent|
next
[-]
yeah, definitely, in the same way that we all regularly go and look back fondly at our chatgpt ghiblified family photos
reply
upvoteby altmanaltman1 hours ago|
parent|
prev|
[-]
Very few things are universally hated. One can love something truly that is hated by most. But it doesn't change the fact that it's still hated by most. An objective and a subjective opinion can exist at the same time on this.
reply