There's something wrong with the Fable 5.2 version, the Customizing button doesn't work for me in Chrome on Desktop.
replyThat's me versus fable. I only told fable to build out the first page rather than the full app that's in the designs (was approaching the limit of my claude plan).
replyI wonder if any of the LLMs are smart enough to verify their work the way humans model 3D objects, i.e. by superimposing the original design on top of their work and checking for differences.
replyThis is actually something I'm working on as part of a more formalized version of this benchmark, both as a tool to assist LLMs in doing this (especially for ones with a more poor vision model), but also to properly measure how well they've done.
replyStill in progress, but early demo showing difference zones: https://non.io/video/fidelity-lens.mp4
I don't know about 3D, but I have been extensively using Fable (with Opus as sub-agents) for https://trailmaps.app [1] and it routinely checks for regressions by rendering maps before and after a change and looking for pixel-level differences.
replyThat, and as I used it to work on the transit-style line drawing plugin [2] it would use the same technique both to test against regressions and look for improvements in the algorithm.
So, yep, they sure seem to be.
[1] Map generator itself is at https://github.com/c0nsumer/trailmaps.app-map-generator
There's some very obvious mistakes in the Fable 5.2 version like the "Start Customizing" button overlapping under the colorful border lines, and the "EXPLORE WITHOUT LIMITS" title being cut off at the bottom.
replyThis would've been caught by a junior web developer or just an everyday Joe glancing at the page for a few seconds
I guess AGI remains two weeks away..
I've been surprised at how long it's taken for this particular test to become fully saturated. So far there hasn't been a model that's able to perfectly one shot it.
replyThis just proves that LLMs are way more jagged than humans and are a tool...for now
reply