If you are working in a company and using language models, it is a very good idea to hold a bunch of evals you can trust and use to validate new models. Calibrate every once in a while with prod data. We have our own and the only numbers on quality and cost I trust come from this setup.
replyA wise man once said, "not everything that counts can be counted, and not everything that can be counted counts".
replyIn the end, the only benchmark that matters is your own.
replyOnly useful benchmarks are those you (in particular) don't have access to.
replyThe only useful benchmarks are those you've created for your specific workflow. Only then can you assess whether a given model is better or worse for what you are using it for.
replyThere are tools like promptfoo designed for this.
> but then we just need meaningful benchmarks that clearly show that!
replyThat's the rub. AI benchmarks are IMO, by and large totally unreliable. We think of them as similar to traditional benchmarks of deterministic processes where the number of variables is low. But they're anything but that. Non-deterministic processes with an astounding number of variables and fuzzy acceptance criteria.
It leads to results like these, where if you take it at face value, the only conclusion you can draw is "wow Anthropic must be stupid if Opus takes 1T parameters to do what Qwen can do in 27B."