upvote
This is actually something I'm working on as part of a more formalized version of this benchmark, both as a tool to assist LLMs in doing this (especially for ones with a more poor vision model), but also to properly measure how well they've done.

Still in progress, but early demo showing difference zones: https://non.io/video/fidelity-lens.mp4

reply
I don't know about 3D, but I have been extensively using Fable (with Opus as sub-agents) for https://trailmaps.app [1] and it routinely checks for regressions by rendering maps before and after a change and looking for pixel-level differences.

That, and as I used it to work on the transit-style line drawing plugin [2] it would use the same technique both to test against regressions and look for improvements in the algorithm.

So, yep, they sure seem to be.

[1] Map generator itself is at https://github.com/c0nsumer/trailmaps.app-map-generator

[2] https://github.com/c0nsumer/maplibre-gl-lanes

reply