I feel like there is a lot of room to build great voice-based agents that don't exist right now.
I have found that ChatGPT voice mode is unusable (e.g. hallucinates me saying things); Claude voice mode is usable, but very buggy around tool calling, and it often mishears things. And it only supports Opus, not Fable (though it looks like you don't support either of those). But I use it anyway.
Question, do any of your TTS options support increasing the speaking speed?
You are right about human input for naturalness, that one we did not automate away with yet. We run blind A/B listening rounds with native speakers.
https://s-1.vercel.app/posts/why-openrouter-can-be-the-next-...
On fast dumb models answering while a smarter one takes over: we are experimenting with exactly that split - a small fast model holds the conversation while a larger one works behind it. Today it runs as two pinned routes, not one packaged API. Most turns in a phone call do not need a frontier model, and the fastest models on our LLM board are all small, so this is where routing earns its keep. We publish benchmarks on LLMs here: https://benchmarks.speko.ai/llm
production phone agents are a different shape today: the call terminates server-side, three models plus turn-taking under one latency budget, and per-language quality still swings a lot from our tests
I was doing it for Veterinary (ambient recording -> SOAP notes) which has tons of complex domain-specific language AND it is critically important to get right.
“CPR” transcribing as “see pee are” just doesn’t cut it in that industry.
We often find that models that wins on clean speech are often not the one that wins on your terms, so test with your own vocabulary rather than a headline number.
Wider picture: we rebuilt the speech-to-speech test around a hard 20-turn concierge call, and today only the closed models finish it cleanly. Board: https://benchmarks.speko.ai/s2s
shared some write-up: https://benchmarks.speko.ai/blog/can-s2s-replace-the-cascade
And jakswa's Gemma datapoint matches what we measure, the fastest rows on our LLM board are the small models like Gemma 4 is performing incredibly well for voice agents
Or even something more managed like Vapi?
Second difference is where it runs. Our gateway is open source and runs in your own container, including with self-hosted livekit/pipecat. You get a temporary token before the session starts, and then your orchestration connects directly to the provider.
Vapi is a managed platform: you use their infra to use the voice AI stack. In our case you can have your own infra and switch between models, so you are not locked into a vendor. A lot of teams we talk to build their own infra as they mature, and that is where the router comes handy.
openrouter is the openrouter for audio models.
the conflation is "audio models" vs "voice ai", and the mental model that untangles it: think batch requests. text in, audio out (streamed, even). audio in, transcript out.
three questions inside the word "router":
1 - what gets picked: a model/voice, a provider of the same model, or the whole stack (stt + llm + tts) per call
2 - where it lives: an external http gateway, the agent platform (vapi/retell/livekit configs), or inside the live session
3 - when: session start, or mid-call
a voice agent is not a batch request. it's a live duplex session: turn-taking, barge-in, telephony legs, session state. the latency physics diverge too: a middleman hop in the media path is paid once by a batch request and on every conversational turn of a live call, so the media path wants a direct connection to the provider. a gateway that terminates at http can route the requests inside a call; it can't route the call.
This isn't Stripe payments. The market will have lots of competitors.
Still shoring up the details but you can try a sample of all the options and see how they compare in terms of model size, peak RSS, real time factor etc.
Linux on desktop is great for you, but this is a tool for people delivering solutions.
That's their dream. Your dream. The AI dream. Many would say it's AI psychosis.
2) you know nothing about me
3) of course I am not! I do know better
for some languages CER is more relevant than WER. Thai and Mandarin have no word boundaries, so we score them by character, and Japanese gets a reading-based CER.
Common metrics to track how coarsely or finely accurate voice AI is
Wondering if you also support some non realtime models.
To use a claudism, I would like to push back on this. The industry is very much moving towards one-model-does-all end to end trained similar to LLMs and VLMs. Mostly for latency reasons and partially because the results for the end to end trained models are just so much better than those using three pieces architectures.
I think most of the value prop is in automatic evals, not routing specifically. A better pitch for you would be "the LM Arena of voice models" rather than comparing yourself to openrouter because the value add is rather questionable. For TTS specifically, the current SOTA for production systems are all using prompt based voice gen i.e. instead of having 10 different Tacotron models trained on 10 different models, these days it's all a single large model and the "style" is a prompt in the system prompt. The input is usually something like
<System prompt>
Speak in a deep smooth voice similar to a documentary narrator
</System prompt>
<Text to Narrate>
Speko is the ultimate evaluation platform for voice agents. We do automatic evals.
</Text to Narrate>
So something like OpenRouter for voices offer questionable value given that stakeholders usually make this sort of decisions once at the start of the project. On the other hand if you can offer automatic evals and figure out which prompts give the most similar results across different voice providers, that would offer a lot more value. It would be nice to be able to switch from e.g. Grok voice agents to ChatGPT voice agents knowing that the output style won't change too much. There are many companies now with evals as a core business model: LM Arena, Artificial Analysis, Prompt foo (before they got acquired and pivoted to security only) so many take a look at them.
Source: we have been building TTS systems for over a decade too https://narrationbox.com
On "promptfoo of voice models": that is closer to how it started. At my last company we ran these evals manually, we would even hire native-speaking raters, benchmark, switch if it wins. The evals are the value, agreed. The routing is what makes them actionable: teams told us swapping always looked like an R&D project, so scores alone did not change what ran in production.
On prompt-based voice gen and reference-audio cloning: agreed, that is what we see too. It makes continuous measurement more important: the same style prompt behaves differently per language and per content type, so we rank the voices themselves, tagged by use case: https://benchmarks.speko.ai/tts-voices
I've worked with hundreds of enterprises on voice AI and voice agent solutions. In my experience, this isn't true. Or rather I should say, the people actually paying for voice agents (i.e enterprises) are not moving towards STS solutions in a meaningful way. The composability, observability, and reliability profile of STS systems is not amenable to enterprise criteria. Not to mention costs.