You're right and that's why we built the router to be cache aware! Once it starts using one model, the threshold to switch to another model will be higher because the additional cost of the cache miss needs to be worth the cost savings or quality increase.
replyThis is the key thing that other routers we've seen miss: they're stateless so for a coding agent use case you end up spending more money due to all the cache misses.
That is interesting, sounds like in practice you only end up routing between 2 models
replyI'd say that a typical main agent loop has 1-3 models (obviously very situationally dependent), but when you have subagents those can get routed independently since they have a fresh context window, so there are a lot more degrees of freedom there.
replyOr not routing at all.
replyIn practice you just pick one and stick with it until the API stops or you hit performance issues.
The choice on the first turn is super important for this reason! But if a user prompt sends the convo in a very different direction then often it does make sense to reroute at that point.
replyThis is a key point. I don't know if you can still edit your submission, but I think this would be helpful to mention up front. I'm looking forward to testing this.
replydeleted
reply