Hallucinations are very damaging to a model’s utility. But doesn’t the Omniscience Index focus on knowledge-based queries? To me, using LLMs for their memorized knowledge is very 2023 and suboptimal.

IMO, what really makes a model useful is its ability to process information within its context reliably and faithfully. I don’t care if it hallucinates George Washington’s favorite color, but I do care about it hallucinating the results of tool calls.

I don't agree. The main issue with their scoring/methodology is that the numbers make it seem like 5-6 models have little to no difference when in fact there is a significant difference between fable and opus and sol and astra for example. They are popular mainstream but most of their benchmarks are either not a representation of model strengths enough or they are not doing a good job of showcasing it properly. The fact that muse and 3.8 were high a day back shows they are just the modern version of lmareana for the mass audience and PR stunts.

This matches what I've seen building anything that uses an LLM for narrow structured output rather than open ended chat, things like classifying a diff into a fixed set of categories or summarizing a change. The aggregate benchmark score barely predicts how it behaves in production. What actually breaks a pipeline is a confident wrong answer on the small slice of inputs that don't fit the pattern it saw during training, not a lack of raw capability. A model that says it isn't sure on the edge cases is far more useful to me than one that scores higher on average but never admits uncertainty, because the wrong but confident output is the one that slips through review unnoticed.