Once you are inserting models into non-creative (and software development is creative work) it is imperative to start creating your own small evals. Not just to check how different models perform in that equation, but to catch model drift over time.
Nope to both. Not even close.
The only way it works is if you define "task" in a way that OpenAI is good at it and Anthropic isn't.
Second of all, barring more cuts, yes I do think the $20 and $100 plans do deliver more tasks because the models have gotten cheaper in the meanwhile (like this very announcement).
> Are they delivering 50% more than Anthropic per token?
So say the public benchmarks. Pull up artificial analysis, “Cost (USD) to run all evaluations in the Artificial Analysis Intelligence Index”:
Sol 6.1 (max): $1,082 Opus 5.5 (max): $8,708 Sonnet 5.5 (max): $7,260
>7x difference in cost for the same tasks.
The original plan doesn't exist anymore. To get the same usage you now need a $500/month plan. The new plan that costs $200 is different in every way that counts. It was actually a 150% price increase for people who want or need what the old plan used to provide.
Don't get me wrong, I genuinely LIKE OpenAI. It's probably worth it even at the new price. I just hate the rhetoric designed to Goodhart past the math.
> the models have gotten cheaper in the meanwhile (like this very announcement).
That is not good news for plan users. Plan users now get less of something that also has less worth to begin with. It would only be good news if they got more of it.
For example, if I buy the "unlimited car wash" package for $19.99/month and the car wash lowers the price per wash from $10 to $1 my subscription becomes less valuable, not more. The breakeven has moved, so I now need to wash my car more than 20 times before the package is justified.
If the car wash then increased the price to $50/month would it be wise for me to keep the package? Would lowering the price per wash make the package more valuable to me somehow? No, it would do the opposite.