Yes, almost all work people share which seeks to measure the capabilities and differences of models needs to get more precise. We are clamoring to say something meaningful about these things.
replyBut even that isn't the whole story because the models can produce wildly amount of thinking output as well as regular output for a similar query. Sometimes you can take a cheap model and have it think a ton or an expensive model that thinks little and get similar results. But the number of tokens generated will be wildly different.
replyA better metric is price per byte. Most thinking traces, prompts, skills are in plain English, which is roughly 1 byte per character, assuming UTF-8 encoding (even code should not be much more either). As an aside, it is common to use bits-per-byte as a loss metric instead of the per token calculation, precisely because of the effect of different tokenizers.
replyIt's going to vary dramatically based on which text you put in. Really it's hard to make one benchmark number that's relevant to all cases. But maybe we can make something a little more specific, like regular English text, code, the model's own thinking tokens, image inputs etc.
replyIt is kind of a shame we ended up comparing token pricing across models and providers when it doesn’t really make sense. Not sure what would be better though.
replyUse price per page (standard English text)? That would also help make the metric easier to visualize.
replyIf you think a page is too vague, use a famous known writer's work as a reference.
Well isn't that what benchmarks are for? They compare total cost for a unit of work.
reply