upvote

points

by km1446 hours ago |
comments
upvoteby booty6 hours ago|
next
[-]

    "benchmark margins have become a less 
    reliable guide to real-world differences" 
    sounds like a big problem.
My guesses:

1. Real-world use cases typically involve big, hairy, crufty, tech debt laden codebases and benchmarks do not.

2. AFAIK "success" in a benchmark essentially boils down to "do the tests pass and do we get the right result?" which is something the LLMs have been achieving with ease for a while, except maybe for uber-challenging coding tasks that would be outliers in just about any workplace. Whereas real-world software engineering is usually just a bunch of CRUD... and "success" involves harder to measure dimensions like "maintainability" and "did you overengineer this?" and "how did you cope with a bunch of vague and maybe contradictory business requirements?"

Having said all of that, I have never ever looked inside any of these benchmarks. I'm putting my guesses out here strictly in the tradition of "the quickest way to learn about something is to be wrong about it on the internet."

reply
upvoteby simianwords6 hours ago|
prev|
next
[-]
It’s likely that they have internal benchmarks but they are communicating to people who can only gauge through external benchmarks.
reply
upvoteby suddenlybananas6 hours ago|
parent|
[-]
Why wouldn't they report these benchmarks?
reply
upvoteby CPLX6 hours ago|
prev|
[-]
Opus 5 fucking sucks. Like it's horrible. I use Fable for coding and anything important and I use Opus 4.8 for things like recursive email categorization, transaction matching, and other stuff where I don't want to burn as much quota.

In my experience Opus 5 is the worst of all possible worlds, it's dumb and headstrong. It just runs away with tasks you didn't ask it to do, is reckless, and basically is unusable in my experience.

Not sure why but my guess is that this will be worse. Happy to be proven wrong.

reply
upvoteby port30006 hours ago|
parent|
next
[-]
I believe Opus 5 isn't meant to be spoken to by humans. It's great at executing but I reckon it's intended to be spoken to by other models such as Fable. I use Fable as the orchestrator, only speak with Fable, and all implementation, recon, design etc happens with Opus 5, with Fable reviewing (and translating).
reply
upvoteby booty6 hours ago|
parent|
[-]
That's interesting.

I've really gone in the opposite direction: having a dumber model orchestrate. In my case, it's usually a Luna orchestrator spawning Sol/Astra subagents to do the "big brain" work of planning and reviewing.

Reason I went with "dumb orchestrator" was just to save tokens. Having Opus/Sol (let alone Fable/Astra) orchestrate was burning tokens like crazy for me even when much of the gruntwork was being done by Luna/Sonnet/Haiku subagents. (Luna is also really good, like way better than Sonnet...) Perhaps it was a skill issue on my end though, maybe I wasn't just managing context properly.

reply
upvoteby cbg06 hours ago|
parent|
prev|
next
[-]
I use it frequently with a lot of success on "Medium" effort, it overthinks like crazy on higher levels, but YMMV.
reply
upvoteby Syntaf6 hours ago|
parent|
prev|
[-]
Yeah if anything Opus 5 taught me how little benchmarks mean to the actual real world performance of these models.

"Better" in every sense of the benchmarks and absolutely horrible results in my day-to-day work.

The verbosity, goal post moving, tendency to leave work unfinished, over focusing on unrealistic root causes when debugging, etc... etc...

It was the first time I actually pinned my models back because I just could not work with 5 for the price and performance it gave me. Hoping 5.5 is better this time around....

reply