Prompt was "read and update the config file with new data". This work on 4.6 takes <2 minutes to read the file, parse the new data, and patch.
Opus 5 Result: 43 minutes of pulling containers, running sandboxes, creating testing suites, which included evaluating the entire repo beyond the scope of the config file.
Both: one file modification
/on The prose is load-bearing unbearable — every sentence feels like it was engineered to sound profound rather than to be read.
I remember how I enjoyed agents between December and February, something started changing around March.
I thought models are getting dumber, but benchmarks were convincing opposite, initially I thought maybe they're quantizing models for day to day use, but Opus 4.8 and Opus 5 seems worse models than Opus 4.6
I think we just need to decouple "doing better on benchmarks" and "actually more useful to me", since they've clearly diverged
The economics catching up with the providers in regards to how much compute they can burn per request and have it make sense for them financially?
A sort of model collapse where Opus 5 seems to love throwing out long paragraphs of text and it needs to be "fixed" by changing the output style and other patches.
I'm not sure, it might also catch up to Kimi K3 and GLM 5.3 and the models that I'm moving to from Anthropic.
>Opus 4.8 and Opus 5 seems worse models than Opus 4.6
After all we've heard about benchmark cheating, I'm earnestly not sure which or whether benchmarks are reliable anymore. But, beyond the models, I wonder if changes to their harnesses and/or instructions dumb them down. I have noticed models change their behavior, even when using the same version/effort. Sometimes for better. Sometimes for worse.
And, I have noticed a model go from really good to struggling. On 4.8 things were going well for a good stretch, so I did not switch to 5 when it came out. Even after hearing complaints about 5, 4.8 was still going well. Then, suddenly over the last few days, 4.8 seems to have nosedived. It feels similar now to the complaints I hear about 5.
In my case, it suddenly started ignoring my design guide, and introducing new fonts etc. It would even use several different fonts and sizes, as well as different margins for similar elements within the same page. It abandoned classes and started inlining styles. It started feeling random and, even after it realized it needed to go back to the design guide, it just continued with more of the same.
There seems to be something that happens after new model releases in both quality and behavior of previous models. It may not be immediately, but eventually there is frequently some regression.
I was a 4.6 acolyte from April til the fable drop, lost that quick, cancelled and took a break, came back a month later, tried opus 5 and liked it, so unpinned 4.6.
Results were great at first, and they're still not terrible, but I have noticed a regression in accuracy, so to speak, where I am pointing out issues that are quite obvious in review.
I pretty much use sonnet 5 low/medium when I have a plan to solve a simple problem and depending on scope, opus low/medium for more complex/bigger scope implementation, and only go high when it's very complex or I'm spitballing architecture/solutions and iterating plan. Never go xhigh or max.
The verbosity is insane though, opus 5 documents everything and just regurgitates whatever lead it to the design choice in there, which makes it more opaque because it's talking about something that was discussed once in a session that no one else can see (except their backend ofc)
I don't even try to steer it away from that with harness, because it doesn't work and just ends up agonizing over whether it should write some comment. Three paragraphs waffling on that on verbose output
I did however have it write a script that basically is git add -A -p for comments though, haha.
I'm $20/month, have all my telemetry toggles off, don't really over engineer prompt/context, just some basic skills for repeated patterns.
It was the wrong time for the GP to drop that subscription from $200 to $20, because $200 gets you a metric assload of cognition while $20 gets you nothing beyond what a local model running on your own graphics card can deliver.
Consistent, predictable behavior is valuable, even more so given the nondeterministic nature of LLMs. Nondeterminism combined with unpredictability might as well be randomness.
I'd guess you're deliberately exaggerating here, but still. I've never clocked the actual tokens/second, but I'm on the $20 plan and get ~15M tokens/month for fully utilized weekly quotas (checked couple months ago). Meanwhile the best I've been able to get locally was ~8 tokens/second with Qwen3.6 35B A3B, which is wildly painful for coding sessions and gets a maximum ~20M tokens in a month... if it's going 24/7.
Just wanted to stick some empirical data here, given that statement.
At $1500 that's 75 months of $20/mo Claude which are MUCH better models than you can run locally.
The point raised by this very article is that you can't depend on that. It's Flowers for Algernon As A Service.
it really angers me every day
You ship 3 hours in three with AI. The old number is meaningless now.
Opus 5 is a neverending chain of "Don't do that. Why did you do that? I've told you several times not to do that but you keep doing it."
"Thinking" for more than 10 minutes for every menial question.
And the prose it writes is horrendous, as if you're reading LinkedIn scammers. "The harsh truth! Two roads, one decision! Reality check!"
I keep finding myself typing “stop overcomplicating everything” multiple times a day as well.
Recently got approved at work for ChatGPT Pro so I could use Codex.
Blown away by the speed. It feels like using Claude Code for the first time again. I don't think Codex is doing anything revolutionary, just better handling of which requests should go to which model, and having faith in some of the "less powerful" models for more than you would think.
It seems the TUI coding experience is very much an open race. This is motivating me to look at other agents / harnesses as well (maybe Gemini, OpenCode, etc).
We sometimes test API serving configs in Claude Code before rolling them out, and one running now maps the numerical effort value differently.
That's why Claude may tell some of you it's at "10" on high. The scale isn't 0-100, the number isn't meaningful on its own, and the effort you selected is the effort you're getting. We've run in-depth evals to confirm this doesn't affect model performance.
This should be the same experience, but if you see a clear regression please hit /feedback and send me the ID. Will give credits.
If I have a user input and then sanitize and inject that into a prompt to do something, I have no idea how much that is going to cost at all and no real way to measure this properly. A parallel example is digital ocean or aws, i can go and measure/limit my compute/fs/memory/startup times/etc and while it can be impossible to get down to the last flop of money allocated - i can run things on a real budget with real constraints, opposed to an LLM where I have to .. prerun a sanitized user prompt through a tokenizer and then ask an LLM to guess what it may do and give token consumption estimates and then act on those in any sane manner for the user?
Perhaps i'm missing something to do realistic and static rails on things but I don't see a serious way at scale to use the token billing model handling things requiring a users free text input short of having to go pander to VC money to throw money at it until someone else figures it out.
*to clarify my rambling... We should be billed and given controls based on resource usage itself and not an opaque token concept on top of not being able to spin any knobs that control it's resource usage.
The model providers are quite aligned with concerns like customer retention. These arguments only work if there is no competition. We exist in a marketplace of black boxes. There's not just "the one" you must suffer. You have options. You can build your own too.
1. Self hosting
2. Chinese models
3. Running it locally. Requires upfront cost and compromises on TPS.
I see posts all the time on HN about which models from which providers offer the most bang-for-the-buck, and how to minimize token usage and still get optimal results, so it appears that competition is working.
Hell, I use 3 different providers, and I currently don't give a dime to Anthropic or OpenAI.
Theoretically.
In reality, one sessions output tokens become the next sessions input tokens (at least if you continue the topic) so, its not as aligned as all that.
But the parent is right, when incentives are not aligned, friction will happen. Its inevitable.
I agree that incentives are misaligned but there’s several competing model providers. If one gets funny with their costs people will jump ship, especially if the gap between the top 2 labs and everyone else keeps shrinking.
These safeguards already exist when they get a whiff that you might be using Claude to fix security issues. Doesn’t seem farfetched given the incentives I outlined that they would apply to this kind of abuse.
How loose those controls are becomes a market force.
https://code.claude.com/docs/en/admin-setup#set-up-usage-vis...
When you’re doing this 100K times per day you get an extremely good idea of what it costs. You also have all the tools to see when something starts changing quickly.
This change is for Claude Code the harness. If you’re using the API at scale and paying full price then you get exactly what you put into the request.
"We sometimes test API serving configs in Claude Code before rolling them out, and one running now maps the numerical effort value differently. That's why Claude may tell some of you it's at "10" on high. The scale isn't 0-100, the number isn't meaningful on its own, and the effort you selected is the effort you're getting. We've run in-depth evals to confirm this doesn't affect model performance. This should be the same experience, but if you see a clear regression please hit /feedback and send me the ID. Will give credits."
LLM doesn't seem to be keen to put in effort either!
Is this AGI?
The chat-based models are obviously being lobotomized based on personal usage and general load (e.g. PST business hours are worst).
API doesn't seem to be affected by this.
In that case, the agent will respond incorrectly because it has no visibility into what reasoning mode it’s in.
but the incentives of these companies to become profitable at any cost slipping into entire new types of dark patterns around token based billing seems gross
- charging for injected prompts and cot tokens
- changing default thinking effort to be higher
- training models to give longer winded answers that don’t say anything more of substance
- refusing to fulfill a request and still charging you
i wonder if you could ever just charge based of each user message and it so how breaks even across short and long replies
EDIT: my honest opinion; Anthropic is building a person, whereas everybody else (it seems) is building a tool.
I realized I was spending most of my tokens arguing with Opus and trying to get it to let go of stupid, lazy, obviously incorrect premonitions. I wound up canceling my 200, bought a pair of Sparks, and am running full fat DS4 Flash and so much happier. Done with being at the whim of these companies.
I have been using GLM-5.3 in my home setup and it is very good in comparison.
I didn't really care about their mobile app and the worst experiences were sometimes the sub-agents within OpenCode freezing and refusing to report their status (though this also happened with Kepler by the GitKraken folks).
I maintain a Claude subscription for Fable but seldom use it.
I was approved.
3 months later, my approval was degraded into "in review" (revoked). I'm sure my account was flagged based on contents of debugging/researching firmwares/etc.
I opened a support ticket. No response. I opened another support ticket. No response.
1-2 weeks later, I got a response that I will not be re-approved and I need to reapply. No problem.
The page to reapply on does not allow me to re-apply because it my account is stuck in an "in review" status.
https://github.com/anthropics/claude-code/issues/84352
The community thinks it's a bug. I'm 95% sure it's not and a bunch of us who were previously approved had it revoked due to flagged content and will not be reapproved.
I switched to Codex + got TAC approved instantly and have not looked back. It's a shame. That's 100% separate from whatever the heck the quality of Opus 5's outputs are. The way it talks... insane. I would bet a good amount of money their next release will focus "reduced simplified responses" if I had to guess.
$2t company by the way
Meanwhile the Chinese models are "go ham dude"...
If it was not for capacity issues, Chinese models have a higher change to just dominate.
> $2t company by the way
It used to be that OpenAI and Anthropic had such a moat around them, that such a valuation was worth it. But these days, its gross overvalued (like so many).
The more stuff is being pulled like cyber verifications, downgrading effort levels, downgrading usage (OpenAI), the more people move to those Open Weight Chinese models.
A fun recent event ... https://opencode.ai/data/
When DeepSeek Flash 0731 came out and provided a massive jump in cheap inference capability. It resulted in a 10x increased OpenCode token usage.
It took a 2.5x to 5.0x price increase AND a reduction by 4x usage (later to 2x) usage, and several cheaper models + a free model, to push the traffic down.
Traffic towards open weight models is increasing, even if providers can not keep up with the influx of new customers. This is not something you want to see as two companies, trying to go for IPOs.
So the idea of stonewalling cyber capabilities, when the rest of the world is just doing whatever with open weight models, on their own hardware even! This entire strategy from Anthropic never made any sense.
Not convinced here.
The problem is that Anthropic seems to be getting away with selling one thing and delivering another. You pay for Opus, you get something else etc.