upvote
> I'm pretty sure that's not the issue. For example I had one problem where Opus struggled for several weeks without getting us anywhere, flip-flopping in its decisions, generating huge amounts of irrelevant hacks.

Sounds to me that you're describing a model trying to navigate ambiguity.

> Fable made it work correctly in 2 days, it stayed on point, identified the real problems, and gave me clean code that was suitable to merge.

If the problem lies in identifying the real problem, that sounds like this is a issue in how you setup the problem statement, aka provide context.

I mean, there are many reasons why in the early days of LLMs and AI code assistants we saw some people using the likes of gpt4.0 to enable software factories while others swore the models were unusable.

reply
There was nothing ambiguous about the task. It had a clear goal and a clear gate to assess success, a gate that takes less than 2 minutes to assess if run correctly (which it didn't until I forced it to encode it as a program).

Your argument that holding the model's hand and telling it what to do and how to achieve the task would make it perform better is ridiculous. It's equivalent to not only giving it the answer to the problem, but also staying all day watching it while it re-derives the solution is correct, making it not only redundant but a waste of time and money.

I remember after a day it made me write in the handoff a hard lesson it learnt after burning 500 dollars of tokens: warming up a synthetic path other than the one the benchmark takes does not meaningfully decrease cache misses. No shit Sherlock, 500 bucks well spent.

It's just literally much dumber than Fable is, which is my argument. I'm not saying it's not usable, I'm saying the difference is quite obvious.

reply