A huge proportion of those scores are gamed anyways. Use whatever works for you at the price and availability you can afford
replyOnly comparing on SOTA scores (ignoring price etc.) is like choosing your daily-driver by looking at who makes the fastest sports-car...
replyThe constant improvements of SOTA are the main thing keeping the investment machine running. We can't really remove training costs from inference costs, because a bunch of the funding and loans for the inference hardware only exists because the promises the continuous training (tries to) provides.
replyNot really. SOTA vs non SOTA is "can I get my coding work actually done today" vs. "this can do customer support chat"
replyIt is like car vs. kick scooter.
> "can I get my coding work actually done today" vs. "this can do customer support chat"
replyI think you need to define "can get coding work done" for this to make sense. Ive been using GPT-3 back-then for basic scripts, does that count ? Or only Claude-Code ?
I also think this is a false dichotomy, if you look at the Project Vend project or Vending-Bench, customer support etc. is at no means trivial. (Old but great story https://www.businessinsider.com/car-dealership-chevrolet-cha...)
It really isn't. We get coding work actually done today on Opus 4.5. That's not SOTA any more, and anything proximate to that level, even quite loosely, is genuinely useful.
replyOK we are in Opus 4.5 is not SOTA. Right by that definition .... yes you are right.
replyI mean its almost halve a year, i think that counts ?
replyOr there will be DSv4.1/2/3 ;)
replyDefinitely something in this realm, they call the models "preview" at a bunch of different points in the paper.
replyWhat im really hoping is for a double-punch like with V3 -> R1
[dead]
replyWell, if they distilled once…
reply