Still fun to watch models trying their best :). I think "money spent" metric in this test is becoming the most interesting to watch. It is like looking at RT cost of "Hello, world!"...

Pelicans, maybe, but the point is to measure how good the "internal visualization" abilities are. Throw curveballs, like a unicorn with a duck bill serving coffee at a basketball court. An elephant playing a piano while its trunk swings a baseball bat at a tiny alien spaceship buzzing its head.

Have them use tikz instead of svg, or have it write code that moves the cursor and draws the thing in paint.

Compositionality and visualization are generally much, much better at each new generation / release cycle.

It's fascinating how well models have internalized visualizing things without actually having joint embeddings / broad multimodality.

I think it's still useful in a "hello world" sort of way. It means you actually tried the new model.