undefined

points

[-]

OpenRouter randomizes which provider gets your request by default right? I think you have to pass a specific provider in the request to prevent that. (Or set up a preset or something.)

This behavior makes it so you don't benefit much from the caching, unless you pin it to a single provider.

by Implicated6 hours ago|

parent|

[-]

> OpenRouter randomizes which provider gets your request by default right?

I'm not sure it's wholey accurate to say they "randomize" the provider, rather my assumption based on usage is that it's something like cheapest-ish/responded to the request within some reasonable-ish time/etc algorithm that chooses the provider on each request - which seems, remarkably questionable in terms of optimizing for user experience or hidden user costs.

> This behavior makes it so you don't benefit much from the caching, unless you pin it to a single provider.

I so very much recommend this approach. My avenues that automate llm calls to openrouter are setup to make api reqs to openrouter to determine best price/response/etc and then pin the request to that (and, preferably, a fallback if there's reasonable difference between #1 and #2) provider for that session. Otherwise you're going to have a bad time.

I'd imagine this could make things interesting in cases where one provider is offering different quants than the others and openrouter is just swapping you back and forth on a long agentic session.

by fc417fc8025 hours ago|

parent|

prev|

[-]

> This behavior makes it so you don't benefit much from the caching

I don't believe this is correct? AFAIK once it routes you to a provider for a given conversation that choice is sticky unless you hit technical difficulties. (It's more complicated than that, they recently added named routing strategies that you can append to the model name.)

IMO the relevant metric is cache TTL which isn't typically published AFAIK.

by dakolli7 hours ago|

prev|

[-]

This is not true, there isn't even a way to see a cache hit % model for a specific model, that wouldn't make any sense. You are confusing what I'm saying with cache cost, that has nothing to do with effective cache hit %. I'm talking about when you click on a specific provider for a specific model, you can scroll down on the view and see their cache hit % for that model [0].

These cache Hit % are accurate, I've done a ton of testing of this myself. The cache hit % is one of the most important metrics as far as estimating cost. There are many providers with cheap cache reads, but have an effective cache hit % of 30%, making their cheaper cache pricing meaningless compared to another provider who charges more but has a 85% cache hit percentage.

[0]: https://openrouter.ai/deepseek/deepseek-v4-flash-0731?endpoi...

scroll down on the provider/model card and you'll see a field called cache hit %, its different for every provider/model.

I don't use routing on openrouter, I strictly use models with a single provider and no fallback, at least for use with harnesses its pretty dumb to route requests to multiple providers you are busting your cache every other request and increasing costs by 20-50%.

by Implicated6 hours ago|

parent|

[-]

I think you're arguing the same general point that the person you're responding to is. But you're saying he's not understanding - he understands that they report a cache hit % but you can't look at that public metric with any level of accuracy _because_ most people aren't pinning their providers and they _are_ getting juggled around which is bringing that metric down. That's not to say that specific providers might have issues or worse cache implementations - but it stands that if openrouter is juggling the requests back and forth by default then _that alone_ is breaking caches on those requests in huge numbers.

by dakolli5 hours ago|

parent|

[-]

I know what they're saying. Why would openrouter calculate it thay way lol. They obviously dont. Think for a sec, they arent idiots.

by ralusek7 hours ago|

prev|

[-]

> Cache hit %

I thought you had to actively manage caches, do you not?