upvote

points

by mgaunard1 days ago |
comments
upvoteby locknitpicker1 days ago|
[-]
> All the people claiming "opus 5.5 is just as good" tends to make me think some people only do simple tasks or do not care about correctness or quality.

Have you considered the possibility of the problem being in the way you prompt a model and prepare it's context? I've been pushing myself to using mainly the cheapest models and I see them deliver the same results once we provide enough context or lay out a plan.

reply
upvoteby mgaunard1 days ago|
parent|
[-]
I'm pretty sure that's not the issue. For example I had one problem where Opus struggled for several weeks without getting us anywhere, flip-flopping in its decisions, generating huge amounts of irrelevant hacks. That obviously went through compaction several times, but I manually maintained explicit documentation and state of the project for it to work with, and reset the session as needed.

Fable made it work correctly in 2 days, it stayed on point, identified the real problems, and gave me clean code that was suitable to merge.

The task was simply to warm the hot path of an app so that latency was not as jittery for some rare, but important, events.

Fable built a reliable repeatable benchmark, disassembled the code to figure out how much could actually fit in L1, made tactical changes to eliminate bloat, and reliably achieved high coverage with minimal edits.

Opus just tried a bunch of random ideas without a plan, hoping to find something that sticks.

reply
upvoteby locknitpicker23 hours ago|
parent|
[-]
> I'm pretty sure that's not the issue. For example I had one problem where Opus struggled for several weeks without getting us anywhere, flip-flopping in its decisions, generating huge amounts of irrelevant hacks.

Sounds to me that you're describing a model trying to navigate ambiguity.

> Fable made it work correctly in 2 days, it stayed on point, identified the real problems, and gave me clean code that was suitable to merge.

If the problem lies in identifying the real problem, that sounds like this is a issue in how you setup the problem statement, aka provide context.

I mean, there are many reasons why in the early days of LLMs and AI code assistants we saw some people using the likes of gpt4.0 to enable software factories while others swore the models were unusable.

reply
upvoteby mgaunard22 hours ago|
parent|
next
[-]
There was nothing ambiguous about the task. It had a clear goal and a clear gate to assess success, a gate that takes less than 2 minutes to assess if run correctly (which it didn't until I forced it to encode it as a program).

Your argument that holding the model's hand and telling it what to do and how to achieve the task would make it perform better is ridiculous. It's equivalent to not only giving it the answer to the problem, but also staying all day watching it while it re-derives the solution is correct, making it not only redundant but a waste of time and money.

I remember after a day it made me write in the handoff a hard lesson it learnt after burning 500 dollars of tokens: warming up a synthetic path other than the one the benchmark takes does not meaningfully decrease cache misses. No shit Sherlock, 500 bucks well spent.

It's just literally much dumber than Fable is, which is my argument. I'm not saying it's not usable, I'm saying the difference is quite obvious.

reply