Maybe I'm biased from the perspective of a "harness provider", but I think Model routers often have too little context to act as well-informed prompt injection prevention. e.g. it lacks context of where which part of the message(s) originates from and sanitization/safeguards were already performed on the application layer.
Something like OpenRouter's "flag" mode is fine, but usage of auto-redact or auto-block should really only be used if there is significant risk exposure through your harness or otherwise they are a constant source of bugs.
But don't worry, many providers there charge for cache the same price like non cached ;)
And if they are, their compliance team is about to strike them down. The VCs forcing this acquisition do know this.
- Why would you add a penalty of 50 ms at a minimum? And that is not the p95... Just run LiteLLM in house and you dont even really need that.
- Their capacity pools are shared across the whole user base, a massive batch processing by another of their customers and think what that means for your response time...
- So instead of negotiating corporate rates with OpenAI or Anthropic, you would be using an intermediary and topping up the corporate credit card... for a 5% markdown ? Really?
- They can see all your critical corporate data on the in and out
- They present some pink SOC 2 promises but then wash their hands and defer to you and the providers. Its just the Bolt and Uber model the drivers are not our employees....
- They are a man in the middle proxy that is a massive security liability for your corporation
- They have no support for private cloud points
- No geofencing guarantees
- No intellectual property legal indemnification unlike what AWS or Microsoft or Google offers
- Its a provider roulette inconsistent with hosts providing different quantization levels causing random shifts in response quality
- Support via a Discord server...
The only reason they were not shutdown yet by Anthropic or OpenAI is because they have the same VCs, as those two. That would mean said VCs investment would go to zero. Oh and those are the same VCs that own Stripe...
Just setup a private proxy tier using something like LiteLLM, even if you really dont need it. Just code your enterprise apps to have have fallback loops on the core hyperscaler providers like AWS Bedrock or Azure Foundry...
All those features show it's just a model selector and router.
I want to know which vendor/model does best at my extracting-facts-from-text task? Which does best at my OCR-a-text-document task? Which can deal with a safe-for-work beach photo without a censorship system false alarm?
OpenRouter lets me run my tests against openai and anthropic and google and x and bytedance and qwen and llama, with a single sign-up and a single payment.