upvote
Model Routing will always be done better by models themselves. Plus routing loses context making it more expensive and less reliable.

Model Routing is just Bitter lesson. The models themselves will get better at this and frontier companies will simply give that capability

reply
This doesn’t seem obviously true, eg an Anthropic model will never route to Kimi even if it were best suited for a particular task.
reply
I think what the parent is saying is that the model itself has the best context for whether a portion of a request should be routed. The specifics of that routing (e.g., should you route to KimiK2) are something that can be trained, finetuned, or even included in a model's startup context.
reply
Has anyone tried that? I have a feeling that if I put it a prompt Claude would comply. But I am all in on the Claude cool aid.
reply
Sure Claude would comply, but Anthropic has no financial (or other) incentive to optimize this, so there’s no reason to expect it to be particularly good.

It would be like asking the clerk at a Whole Foods which grocery store in the city sells the cheapest eggs. He’d probably answer - he might not even say Whole Foods - but WF is hardly teaching all their staff the best methods to answer this question in training. (Heh, training.)

reply
Why should it? An Anthropic model is architecturally optimized for Anthropic models, routing it to Kimi makes zero sense
reply
Which is why 3rd party routers which do route between different models may have an edge. It means they can compete on cost, and it’s definitely not clear that the architectural optimization is always going to be higher quality or cheaper. It might be, but everything changes constantly, so locking into a single model family/company is very much not ideal
reply
deleted
reply
i think you're thinking of subagent routing

model routing in this case is cross-provider

Imo the main issue behind model routing is you need to figure out how much intelligence a new task takes, which is a very non trivial problem. Presumably, a organization knows this about their own tasks and is better suited to built in-house compared to outsourcing to a vendor.

reply
Model routing by the model itself requires the model to pull in a lot of context and it's likely more efficiently just done by people with the context already in their head, even assuming the model is perfect at routing (which last I checked, Claude definitely isn't). I wouldn't trust ML model routers.
reply
Because they're trying very hard not to understand it.

Otherwise the expensive-yet-powerful model probably won't see much revenue. How much money is there in bleeding edge scientific research? There's a lot, but there's even more existing capital in paying people people to do college level paperwork, and the bulk of those traffic gets routed to the cheapest model.

You mostly don't need super powerful AGI to replace the paper pushers, but the frontier labs are trying to position themselves as being uniquely capable of producing super powerful AGI, and also be the ones replacing office workers.

Not sure how it will work out for them, but I think model routing is going to poke holes in that narrative. That's why I think they're trying very hard not to understand model routing exists.

reply
It wasn’t long ago at all that the chief problem was “can AI even help me with this” (cost be damned). Until a time when the answer is an unmitigated “yes, obviously”, the frontier labs have everything to lose and nothing to gain on routing, because if they screw it up you might incorrectly decide “nope, it can’t help yet” due to a poor routing decision.
reply
Who are the customers though? Honest question, I'd like to understand it.

For me, anything other than current best available SOTA for any task is unacceptable. The only routing rule I need is "the most powerful model I still have flat-priced quota available for". I mean, why settle for less?

reply
There are two types of users: those who are able to use subsidized rates, and those who need to use API rates due to audit requirements, enterprise billing, etc.

Model routing for subsidized users takes the form of a "use Opus 5 subagents for implementation" type of system prompt. You lean into a single provider, build tooling around that, and your savings are far beyond anything multi-provider routing can get you.

Model routing for enterprises is far more complex - approaches like https://fireworks.ai/blog/kimik3-fable become necessary for cost control.

reply
on top of that there is also additional factor: speed - sometimes if task is easy you do care to finish it faster.

There is also matter about convenience - when I ask some small easy question often I don't bother to switch the model or forget in prompt to ask faster/cheaper subagent.

reply
It's very common to use a lesser model for a lesser task, resulting in same quality output. End result: save money while being faster. In many cases, it's a pure win-win.
reply
But in other many cases you have to redo the work directly or indirectly, and you are more expensive (for now) than even the most expensive models, so sounds like a total lose.
reply
I’m assuming you don’t pay per API call. Every mid-large sized business in the world does.
reply
And you do not have flat priced quota for Fable 5, right? Because nobody has, as far as I know. So you'll probably not route any task to the "current best available SOTA".

Also: quota. Implies you do not have unlimited access even for flat prices. Which in turn implies that as soon as you hit the quota on the most expensive flat price plan, even you will suddenly discover the magic of economically sensible behavior.

reply
Fable 5 is included in the Claude Max subscription. I've gotten close to the limits this week and last but haven't hit it yet.
reply
But Fable is only available for 50% of your Max quota as far as I understand
reply
Not everyone is you. Other people probably have a range of tasks that can accomplished with different models.

Certainly if I'm confident that I'm going to get what I need from a faster model, that's what I want to use, rather than wasting time grinding away for the sake of saying of the same answer came from a SOTA model.

Given that every chatbot does offer a range of models, it seems clear people do choose among options.

reply
The mental effort in estimating what model would be better is so not worth it.

I just want to switch to Claude Code, tell it to turn a .csv into a BigQuery table then cmd+tab to something else while it runs. Thinking "oh this is probably an easy task, I can /model to Sonnet to save $0.0004" is silly.

reply
because "less" can be so much faster?
reply
not just that -- "less" ($$$) can also result in indistinguishable quality for some tasks/inputs. I'd argue this is the primary reason, secondary being speed.
reply
I generally agree. Perhaps there's only a 5% chance that it would write better code or find a bug that it wouldn't have with a lesser model, but the economics of bugs is strong enough that preventing a single bug is worth hundreds of dollars.
reply
That's nice and all, as long as flat-priced quota continues to exist. Seems unlikely to go on forever!
reply
> For me, anything other than current best available SOTA for any task is unacceptable.

Then you must route. An article with lots of upvotes yesterday or two days ago showed that K3+Fable 5 was more SOTA than either of those.

reply
I'm not a customer of those routing systems, but I quite often use different Claude models for different tasks. While most tasks were Opus 4.8, I often used 4.8 to make a plan, prompts, and package kit to setup Fable for a bigger project, then run it on Fable. Or, for broad single-task searches Sonnet with or without "Research []" turned on seemed to work best both faster, lower overhead, and less verbose answers (when I didn't want it).

OFC, YMMV

reply
I am not convinced this will end up being a domain of the ‘routers’ vs the clients, as in harnesses themselves. Thoughts?
reply
what's the threshold for model routing where you're willing to trust the router?

For coding my own work I don't trust the model router, and it would have to be shown to be to save a real dollar amount.

From a buying perspective it's a hard sell to save x but lose out on bugs you are probably introducing at an unquantifiable severity and frequency. How much is it worth to hedge your bets by doing every single inference request on the frontier model?

How much will it cost to go back later and fix things, but also the meta question of how to be able to decide on a hypothetical unknowable? (You'll never know how much better or worse your code was gonna be, it's untestable at a project level)

reply
you trust the service provider but not the router?

weird, but ok

reply
I don't want to save money so badly that I'd possibly undercut the quality of the code that gets created.

*edit to add: that code quality (or lack of quality) is it's own cost

reply
deleted
reply
lol I had to get ChatGpt to explain to me the difference between 5.6 sol, 5.6 Terra, 5.6 Luna, 5.5, 5.4 mini, 5.3 spark, and then there is low, medium, high, extra high, max, ultra, and pro… I still don’t really know, it feels like ordering hot wings.
reply
Openrouter should ideally kill in this space and make their model agnostic infra like memory, harnesses, chat applications.
reply
OpenRouter is in acquisition talks with Stripe, fyi

I would expect routers to commodify like tokens.

reply
OpenRouter sprang up overnight. I might need to replace some urls and access tokens should they decide to try and screw me.
reply
I never signed up because I found the 5.5% fee on token usage to be a "screw you" tactic. Still do not understand why they are popular with the other options out there.
reply
And it's not just that model routing is much cheaper: no longer than yesterday we got a post showing that routing between K3 and Fable 5 was more SOTA than either of those.

If that is true, model routing is here to stay.

It also seems to validate the minimalist approach of pi.dev, where sub-agents from the same company is not the preferred approach (pi.dev believes in neither sub-agents all from the same company nor MCP even you can do it if you want for pi.dev's philosophy is to do add any functionality you want to a minimal harness).

Now of course we'll get for a few weeks all the Anthropic fanbois and shills explaining that "sure, K3 was basically at the level of Fable 5 but now that Opus 5 is out, open-weights models are six months behind".

reply