Sounds like a reasonable choice of a language.
replyI myself have written a small llm proxy in Go, doing nothing but injecting the real API key to outgoing requests while itself validating virtual keys it generates, and it works great.
Definitely it was a deliberated decision. Also enterprises adapt Go more and more.
replyIf you don't use any LLM router, it's infinity times faster, and infitesimally as targetted.
replyI have GoModel running at home, being deployed at work. There are many good reasons to run a gateway, some include
reply- being resilient to upstream provider downtime
- centrally managed config/controls
- visibility and allocation into usage and costs
We are deploying a gateway at work primarily for these three.
At home, GoModel's virtual models have been super helpful for testing out new open weights. I can change the model deployed in vLLM without having to change any of my downstream tools, which get a stable name.
deleted
replyYeah, this DIY MITM attack is entirely avoidable.
reply