It's common for different models to find holes in another's work. There are various good reasons for that.
replyFWIW, we use ChatGPT for our primary model and use Claude to do the reviews. This works better than ChatGPT doing it's own review even with a clean session/context.
It's beyond common for a model to find holes in its own work, as well. I have an iterative review as the part of all agentic work, and it always finds something to fix, and will sometimes spend hours fixing its own work.
replyAgree, but the point is not because Fable is better than Sol, it's because it's .. different .. it just looks at the problem through a different angle.
replySame here. Grok Build 4.6 for me, given how cheap Grok is and how Sol is supposed to be "the" SOTA, it finds a surprising amount of bugs. Most of which Sol agrees with needs to be fixed or improved.
replyI've done this tens of times between these two models and it works great in my experience. Sol initial back and forth with me. Commit. Let Grok review. Sol fix. Only then do I start reading the code.
I suspect it would work with the models swapped too, or even with one model and a blank context for the second run.
replyI used to review each others work, Sol is amazing at review and finding what’s missing.
reply> Just the other day I had Fable draw up a solution for me, and then I fed it into 5.6-Sol and said how does this look…
replyYou should be doing this for every solution.
Even Fable reviewing itself will find issues, unproven assertions, etc. Same for Codex models. A review loop is critical.
Did you try to say "think more deeply about this problem" to fable after getting your solution, having one model focused on creation then blaming it for not doing proper review when the other model was told to focus sol-ely (pun intended) on review is not a fair apples to apples compaision
replyI mean, then it sits there stewing for 20-30 minutes when you can ask sol and get the same answer in 5.
replyLike, the quality of the anthropic models is fine, but they’re so incredibly slow. Claude reads files one at a time while codes dispatches tool calls three or four a time.
The fair comparison would be to also do the reverse: start with Sol then have Fable clean up. Then compare the Fable-Sol and Sol-Fable outputs side by side.
reply