> So your flagship model must be fastest and most accurate.

That's rarely the way anything in life works though. Do you prioritize fastest OR most accurate? That's a question that depends on context, task difficulty, etc.

You can use a 17M parameter NLI model for simple stuff and get orders of magnitude faster decisions. But the intelligence of that model will be quite a bit weaker.

In some tasks you might be talking about very small differences in accuracy between that model and a 27B parameter LLM decision model for several orders of magnitude in speed. In other tasks, the 17M might be about as good as a random guess.

But isn't there still a meaningful spread of difficulty, risk tolerance, etc. in decision model relevant tasks? Especially at high volumes I think there would still be value in being able to make intelligence/cost/speed/calibration tradeoffs between models