upvote
If you have the quota or budget for it, I recommend vibe-coding your own personal benchmark. I have a suite of 8 projects that I made with models this year, and replay them - give the new model a checkout of the repo at that point, and the project spec. For the design task, I have it throw up a blind voting page for me to quickly judge, and for implementation I have a review agent do a general score and also hunt for problems/edge cases that I really found.

It's a lot more imprecise than the public benchmarks, but it's tailored to my own work and definitely isn't being benchmaxxed by the model developers. I was able to tell really quickly that Opus 5.5 delivered on the hype, and that some other releases (a couple of the more recent open releases, and pre-Carbon gemini models) were not as good for me as public benchmarks made them look.

(no results for either 6.2 yet, will take another ~6-8 hours, but I encourage people reading this with spare tokens to just make their own like I did instead of listen to me)

reply
This is much more interesting IMO than any public benchmark. They are definitely targeting those benchmarks and model capability does seem to have plateaued somewhat.
reply
What type of tasks did you use for this? Evaluation?

Last time I did this I found it quite hard to come up with tasks that have clear evaluation outcomes

reply
please update !
reply
I didn't find Sol 6.1 nearly as impressive in real SWE use compared to what the benchmarks implied. It wouldn't surprise me if the models have internalised some kind of 'benchmarking mode' which only kicks in when some attributes that are peculiar to how benchmark problems are presented.
reply
We've recently moved away from Anthropic models at $PAID_JOB and over towards GPT models. I'm trying to be open-minded but the GPT models we have (5.6 Terra and Sol) are just not that great. They're fine, but even simple things such as "check my work on this small Bash function" get it to point out stuff that's already implemented in my work. I don't know if it's me/how I use it/whatever, but just some N=1 evidence, I guess.
reply
5.6 is in no way representative, or comparable to 6.1. In fact, the 6.1 models are so good that I've just used Terra 6.1 xhigh for most tasks.
reply
GPT models in my anecdotal experience are worse at drifting into the weeds on long running tasks, and continue to take what you ask way too literally. I have come to the conclusion that Anthropic’s models are just better at teasing out and intuiting what you actually mean even if you under specify it.

They are much better compute for dollar though. I have hilariously started to just have an opus thread maintain Sol subagents to get the best of both worlds

reply
sol 6.1 is meh
reply
why do you guys have to use 5.6?
reply
IYKYK. But in short because of a lack of alignment with product strategy
reply
I have the exact opposite experience.
reply
5.6?? How is your impression of 5.6 relevant to anything recently?
reply
Well, yes, if you switch from current claude model to something that was released in July, of course it will be worse...
reply
Bro you have to be at 6+. 5.6 is EARLY JULY
reply
5.6 series is worse and more expensive than the 6 series. Try the latest Luna and sol; I use those for 100% of my work right now, with sol doing only 10-5 percent of it. I code in go and review the work and edit it extensively though, so I don’t know about how good those two are at vibing slop.
reply
5.6 was better than 6. 6.1 is a bit better than 5.6 from my tests
reply
Yeah, in real SWE tasks, 6.1 is bad.

We've been trying to capture real SWE data by building a battle system on top of OpenCode. In our data, Sol 6.1 is seriously underperforming, I'm talking worse than DeepSeek 4.1 Flash, at 10x the cost. (Though to be fair, statistically they are tied in performance, though not at all in cost. https://agentduel.com/leaderboard .)

(It's possible for a tiny model to do better than a large one because contrary to typical Arena methodology we think a fast good enough answer is better than a slow one.)

OpenAI performance on our system has been bad enough I switched to Anthropic models for my daily work.

reply
I'm guessing you don't use codex though- the harness + model combo makes a difference.
reply
I've been building quite a complex desktop app (a digital audio workstation) using 6.1 Sol and it's been very good for me
reply
Is this an ad?
reply
No. I'm personally building agentduel.com, specifically with this problem in mind (benchmark performance not matching real world usage). It's generally accepted by HN community to make relevant comments about your own projects, as long as its not hamfisted.
reply
Yeah, MaxxBench was shared here a while ago and showed most models think about how a grader will evaluate their work on DeepSWE in 95%+ of traces. Strong models definitely realize they're being graded and start to game their output.

https://maxxbench.unlap.ai/research/why-maxxbench/

reply
I had the same impression. None of the GPT-6 models were as great as they claimed, they never surpassed Fable or Opus 5.5 on my use-cases.
reply
You make it sound as though the models are like not working to their full potential instead of being a little bit overfit on the benchmarks.
reply
> Looks like they did time this for right before Anthropic's IPO.

Would it be in OAI's interest to kick the legs from under Anthropic? Granted I don't think they can, but a shot across the bow can sometimes be more damaging that one expects (I think FTX collapsed due to one? Or maybe I'm misremembering).

I always assumed that if one of the big AI IPOs fail hard enough the neoclouds start to go bust, which causes a chain reaction that probably causes both ANT and OAI to collapse.

reply
My reaction as well - at the very least now is probably not the best time to knock down the realized valuation of a peer when they are effectively setting the market value of the whole category of ai companies, including yours… right?
reply
>Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?

Generate an SVG with complete and detailed graphical instructions on how to fold an origami pelican riding a bicycle

(or any other complex origami object). The fact that it has to teach [you] something makes it pretty interesting.

reply
> Looks like they did time this for right before Anthropic's IPO.

While both Anthropic and OpenAI have been playing games with model announcement timings, this time I don't think it's related.

These are incremental bumps, not big new models that have a huge impact.

reply
They're going after anthropic's market through cheaper intelligence per token.

I wouldn't be surprised if they released 6.2 luna with the most attractive positioning on the pareto frontier right as the IPO happens.

reply
I'm very curious about this as well.

Recursive self-improvement (RSI) has been a very hot topic recently, and it could be the biggest factor that decide Anthropic's IPO.

I wonder if the general public can infer whether RSI has started based on the recent performance gain of the frontier model.

reply
RSI just means using the LLM as part of the development process, which they are already doing. I doubt any of these companies are going to fire all their developers, so there is no point development is going to be 100% automated - for the foreseeable future it will always be a mix of humans using LLMs.

So, you can say they are doing "RSI" (aka using tools to automate development) already, and will continue to do so.

reply
Why IPO if you have RSI and possible ASI?
reply
So investors can cash out on the time frame they promised investors.
reply
Yeah we see some Really Stupide Intervention from those who drank the Kool aid. IAS2[] start to be a real thing.

[] IA Stockholm Syndrome.

reply
I've been having fun with having it build interactive Three.js demos of devices, with setup screen, a few different operational modes, mechanical movements etc
reply
You're looking for a vibe check?
reply
> Feels like we're way past the pelican benchmark being useful at this point.

It was neither a benchmark nor a good one at that. Just a blogger's curiosity that grew out of control.

reply