undefined

points

[-]

Models seem to have very interesting blind spots. While https://assbench.com/ may seem like a joke, the results are very illuminating. One might expect far better results given the detailed prompt, allowance for render checks and iteration, the number of human figures the models are trained on, and clarity of tool use (threejs).

However, result quality really drops off once you get out of the top 5 or so.

(NSFW note: all buttocks are covered by fairly conservative bikini bottoms and show no genitals, bulges, etc.)

by simonw7 hours ago|

prev|

[-]

The models that are really good at 3D modeling tend to also be really good at pelicans. I think the skills are strongly related - it's about having good 2D (and now 3D) spatial awareness.

The test is always going to be hard though, because it's an absurd image to ask for.

UPDATE: I like this theory about using an agentic harness that can render and iterate better: https://news.ycombinator.com/item?id=50073879#50074395

by boo_you2 hours ago|

parent|

[-]

> it's about having good 2D (and now 3D) spatial awareness.

this is a really profound insight

by TeMPOraL6 hours ago|

prev|

[-]

"Pelican on bicycle in SVG" is an adversarial example on multiple levels - out of distribution, ill choose of tool for the task, and anatomical impossibility of real pelicans using typical bycicles.

by charcircuit6 hours ago|

parent|

[-]

But often these models mess up drawing even just the bicycle part.

by duckmysick6 hours ago|

parent|

[-]

I remember a submission a few years ago about a guy asking others to sketch bikes. Most of the time the drawn bikes wouldn't be rideable.

https://www.gianlucagimini.it/portfolio-item/velocipedia/

HN discussion: https://news.ycombinator.com/item?id=17699017

by mason556 hours ago|

parent|

prev|

[-]

To be fair, humans can’t draw a bicycle either.

https://www.wired.com/2016/04/can-draw-bikes-memory-definite...

by jephs7 hours ago|

prev|

[-]

The pelicans are one-shot with not tools: Think a bunch, write a bunch of SVG, done.

The pretty 3d stuff you see are agents controlling a Blender MCP in a loop where they look at the scene, think, adjust the scene, repeat.

by simonw7 hours ago|

parent|

[-]

Yeah, that's a good point. My basic pelican test doesn't allow the model to render their work and iterate on it. The 3D models all come from agent systems that are rendering with Blender or screenshotting with Playwright along the way.

by agar4 hours ago|

parent|

[-]

Have you looked at assbench.com? Even with threeJS rendering, screenshotting with Playwright, and iteration (check the prompt) some of the models are shockingly bad. I wonder whether the rendering quality is affected by the requirement for skin physics on click; if so, it raises a valid question about models creating pretty pictures but break when interactivity is required.

(As I said in a separate comment above, results are all bikini-clad and not revealing. However, whether it is NSFW is up to you/your work policies.)

by epihelix2 hours ago|

parent|

prev|

[-]

If you allow rendering and iteration on the basic pelican test, does it improve the final output?

by vineyardmike7 hours ago|

parent|

prev|

[-]

I’d really like to see what each model comes up with if given a basic harness to render and iterate. They’ve all gotten pretty good at basic one shot pelicans, but seeing what they do when given a direct goal, and even getting a time lapse of each image as they edit and adjust would be really fun.

by AspireOne5 hours ago|

parent|

[-]

Very easy to do. It's a nice silly thing to try, so I'll conduct this very important experiment and update you on the results if I remember to do so

by sobiolite7 hours ago|

prev|

[-]

A lot depends on what the models were specifically trained for using reinforcement learning. Modern models are generally OK and everything, but they're only really good at the things they have extensive RL for.

by ben_w4 hours ago|

prev|

[-]

> Modern models can make nearly photorealistic games in threejs. So why do they struggle with these pelicans?

Are you sure they're not fetching free asset packs for that?

by Perz1val7 hours ago|

prev|

[-]

Have you tried editing SVG by hand?