The roulette wheel isn't rigged, sometimes you're just unlucky. Try another spin, maybe you'll do better. Or just write your own code.
Also, another difference is the stochastic nature of the LLMs. With table saws, CNC machines, and modern 3D printers, you kind of know what you are getting out. With LLMs, there is a whole chance aspect; sometimes, what it spits out is plainly incorrect, sometimes, it is exactly what you are thinking, but when you hit the jackpot, and get the nugget of info that elegantly solves the problem, you get the rush. Then, you start the whole bikeshedding of your prompt/models/parameters to try and hit the jackpot again.
But it's also a tool that (can) save(s) you time.
Absolutely, not understanding why you even ask. Humans are creatures of habits that often dip a bit or more into outright addictions, in one of its many forms.
This scenario obviously does not apply to folks who run their own benches with the same inputs between models. I'm just discussing a possible and unintentional human behavioral bias.
Even if this isn't the root cause, humans are really bad at perceiving reality. Like, really really bad. LLMs are also really difficult to objectively measure. I'm sure the coupling of these two facts play a part, possibly significant, in our perception of LLM quality over time.
I've cancelled my subscriptions to both Codex and Claude and am going to go back to writing my own code.
When the merry-go-round of cheap high quality inference truly ends, I don't want to be caught out.
"I think we can postpone this to phase 2 and start with the basics".
Meanwhile using more tokens to make a silly plan to divide tasks among those phases, complicated analysis of dependency chains, deliverables, all that jazz. All unprompted.
And it does seem likely to me that there were intermittent bugs in adaptive reasoning, based on posts here by Boris.
So all told, in this case it seems correct to say that Opus has been very flaky in its reasoning performance.
I think both of these changes were good faith and in isolation reasonable, ie most users don’t need high effort reasoning. But for the users that do need high effort, they really notice the difference.
We aren't superstitious, you are just ignorant.
Don't use these technologies if you can't recognize this, like a person shouldn't gamble unless they understand concretely the house has a statistical edge and you will lose if you play long enough. You will lose if you play with llms long enough too, they are also statistical machines like casino games.
This stuff is bad for your brain for a lot of people, if not all.
Some day maybe they will converge into approximately the same thing but then training will stop making economic sense (why spend millions to have ~the same thing?)
I have flexibility to shift my core working hours (and what I do during N/A business hours). Knowing they're explicitly making it dumb because of load is important. It allows me to shuffle my work around and run heavy workloads late at night (plan during working hours then come click "yes" a few times in the evening).
Reading about all the “rage switching”, isn’t it prudent to use a model broker like GH Copilot with your own harness or something like oh-my-pi? The frontier guys one up each other monthly, it’s really tiring. I get that large corps may have contracts in place, but for an in indie?
How will your project/decision look on the front page of the Wall Street Journal? Well when a whistleblower reveals what everyone knows ($9b->$30b rev jump w/o servers growing on trees simultaneously = tough decisions), it's gonna be public anyway.
And the andecdata matches other anecdata.
Maybe I'm missing why that's selection bias.
lmao, no they shouldn't.
Public sentiment, especially on reactionary mediums like social media should be taken with a huge grain of salt. I've seen overwhelming negativity for products/companies, only for it it completely dissapear, or be entirely wrong.
It's like that meme showing members of a steam group that are boycotting some CoD game, and you can see that a bunch of them were playing in-game of the very thing they forsook.
People are fickle, and their words cheap.
But this isn't like a minor debacle about a brand. The flagship product had a severe degradation, and the parent company won't be forthcoming about it.
It's short term thinking. Congratulations, everyone still uses your product for now, but it diluted your brand.
Why take the risk when the alternative is so incredibly easily? Build engagement with your users and enjoy your loyal army.