undefined

points

[-]

To keep it simple, forget about routers and imagine you're in Cursor using GPT for a while, reaching a cache of says 200k.

You decide to switch to DeepSeek in the same session via the model picker, and continue as usual. What happens is that the cache for DeepSeek is created with the 200k + the incremental message. After this, cache can be kept warm for both models; two instances of the cache exists, one for GPT and one for DS.

You switch back to GPT. The whole session is sent to the model with the 200k original from GPT and the incremental messages you sent to DS. The 200k is read from cache and the incrementals are new, and then added to the cache.

Let's say every second message you switch between GPT and DS; cache was 200k and each incremental message is 1k. If you kept going with only GPT, cache hit rate would be 200k/(200k+1k) = 99.5%. When you switch between two models with warm cache, hit rate instead becomes 200k/(200k+2k) = 99%.

Model routers work the same way. Keep the cache warm, replicate it in two places. For this reason, when you set up your model pool for routing, you want to keep the model pool small and differentiated.

First principles of model routing: https://try.works/first-principles-of-model-routing

role-model router and protocol: https://github.com/try-works/role-model

note: edited to keep the answer to the below message clearer

by hedgehog15 hours ago|

parent|

[-]

To elaborate, because I don't think some of the people reading this understand the reason, typically a lot or most of the cost in "agentic" API usage is cached read + generation. Cached read costs scale with turn count, which multi-model switching doesn't increase, and of course generation gets cheaper if you do some of it with a cheaper model. When you switch models the "catching up" batch of messages is just a single prefill and then that goes into cache. You don't even need to have the same chat history across models so long as the view from each model's perspective looks like a series of appends.

The main problem with model routing in my experience is that to work well the router needs to be pretty strong, maybe even moreso than any of the actual models in service. There are probably clever solutions to this but I haven't seen any that look better than just using sub-agents.

by richwater15 hours ago|

parent|

[-]

> which multi-model switching doesn't increase

Given model A with cache C(a) and model B with C(b)

Isn't this not true because the moment you switch models from A to B, you need to provide C(b) the latest conversation diff since C(b) last updated, say many turns ago?

by WASDx14 hours ago|

parent|

[-]

If you take 10 turns with a model A, it has to read the cache 10 times and write a lot of tokens (the expensive part). Switching to model B is just prefilling the diff + your new message, which is still just one turn. So total number of turns does not increase for a long session even with many switches.

I didn't understand this before reading the sibling comments so I'm not sure I got it fully right but I think the total cost becomes like this:

* Input tokens: Pay for both models * Output tokens: Pay for the model that generates * Cached tokens: Pay per turn, so in total a weighted average over both models?

Since output tokens are the most expensive, I can see how this is an overall win for many use cases as benchmarks also show. The hard part is routing correctly.

by hedgehog14 hours ago|

parent|

[-]

[dead]

by thehamkercat15 hours ago|

parent|

prev|

[-]

Can you explain how does it work? like how is the previous K/V cache used when you switch to another model?

Source?

by hedgehog15 hours ago|

parent|

[-]

See sibling answer but essentially the effectiveness of cache is not diminished by having a separate one per model (relative to the win of doing more turns and generation with a cheaper model).

by try-working15 hours ago|

parent|

prev|

[-]

edit: updated the answer above to be more qualitative instead