> All flagship models are within like 1-5% of each other

Don't know about that.

I'm using code review of my lone lisp project as a benchmark. It's a massive parallel code review where a coordinator cuts up the codebase into sections and dispatches agents to consider each part from different perspectives like quality, maintainability, consistency, correctness, rigor, etc.

Ran a complete Fable/max code review. Took over a month on a subscription. Now I've switched to OpenAI and am repeating the exact same review with Sol/max.

It's still not done yet but preliminary findings suggest Sol can only reproduce 70-90% of Fable's findings. So I think these models aren't as close as we've been led to believe.

The problem is that most of the volume doesn't come from proprietary products, it comes from API use which has no stickiness.

Claude already has a killer product (claude.ai/chat is a Swiss army knife) but just relying on people typing stuff into chat is not enough to sustain the company.

The other strategy is entrenching yourself as the LLM of choice into existing products (like ChatGPT is on Apple products).

> All flagship models are within like 1-5% of each other

Depends on your use case. the Chinese models are not there yet.

There is a massive difference even between Opus and Fable, same provider, before various harnesses and other optimizations come into play. Don't be deceived by rankings and benchmarks, try for yourself.