I don't have anything to add, but I like how you guys are actually sending people to communicate in Hacker News. Brilliant.
replyyou're one of them?
replyMaybe a good idea to be more explicit about this -- maybe a cost analysis benchmark would be a nice accompaniment.
replyThis kind of thing keeps popping up each time a new model is released and I don't think people are aware that token efficiency can change.
Agreed. Would be great if everyone starts reporting cost per task alongside eval scores, especially in a world where you can spend arbitrary test-time compute. This is one thing I like about the Artificial Analysis website - they include cost to run alongside their eval scores: https://artificialanalysis.ai/
replyTheir subscription subscribers will see/feel the difference irregardless, API pricing is hopefully read by devs that know about token efficiency and effort.
reply