Fair criticisms. If it's comparable with haiku, that's $13/day or 47 days til break-even (again, assuming I have my system generating tokens 24/7 - which I do not).

> You’re also comparing benchmarks for the full model

I wish there were more concrete ways to judge the difference. I've spent a bunch of time using 3 and 4 bit versions of qwen 27b, and my conclusion was that the lower qaunt made more mistakes - but that both systems were able to recover from those mistakes in a coding context. I'm not sure if that holds across the board, but that's the best answer I have so far.

It's also unclear if anthropic always serves you f16 versions of their models. My friends who use claude a lot complain about intelligence that varies from day to day - but that's entirely anecdotal.