undefined

points

[-]

> Remember that you are programmers and you can just program, you don't need a framework, you are already using the API of an LLM provider, don't put a hat on a hat, don't get killed for nothing.

Programming for different LLM APIs is a hassle, this library made it easy by making one single API you call, and in the backstage it handled all the different API calls you need for different LLM providers.

by hrmtst938371 days ago|

parent|

[-]

One wrapper cuts API churn, but it also widens the supply-chain blast radius you own.

by rcleveng22 hours ago|

parent|

prev|

[-]

I think almost everyone supports the openai api anyway (even Gemini). Not entirely sure why there needs to be a wrapper.

by dragonwriter21 hours ago|

parent|

[-]

Msot do, but Anthropic indicates that theirs is "is not considered a long-term or production-ready solution for most use cases" [0]; in any case, where the OpenAI-compatible API isn't the native API, both for cloud vendors other than OpenAI and for self-hosting software, the OpenAI-compatible API is often limited, both because the native API offers features that don't map to the OpenAI API (which a wrapper that presents an OpenAI-compatible API is not going to solve) and because the vendor often lags in implementing support for features in the OpenAI-compatible API—including things like new OpenAI endpoints that may support features that the native API already supports (e.g., adding support for chat completions when completions were the norm, or responses when chat completions were.) A wrapper that used the native API and did its own mapping to OpenAI could, in principle, address that.

[0] https://platform.claude.com/docs/en/api/openai-sdk

by TZubiri1 days ago|

parent|

prev|

[-]

>Programming for different LLM APIs is a hassle

That's what they pay us for

I'd get it if it were a hassle that could be avoided, but it feels like you are trying to avoid the very work you are being paid for, like if a MCD employee tried to pay a kid with Happy Meal toys to work the burger stand.

Another red flag, although a bit more arguable, is that by 'abstracting' the api into a more generic one, you achieving vendor neutrality, yes, but you also integrate much more loosely with your vendors, possibly loose unique features (or can only access them with even more 'hassle' custom options, and strategically, your end product will veer into commodity territory, which is not a place you usually want to be.

by otabdeveloper41 days ago|

parent|

prev|

[-]

There's only two different LLM APIs in practice (Anthropic and everyone else), and the differences are cosmetic.

This is like a couple hours of work even without vibe coding tools.

by dragonwriter21 hours ago|

parent|

[-]

> There's only two different LLM APIs in practice (Anthropic and everyone else), and the differences are cosmetic.

There's more than that (even if most other systems also provide a OpenAI compatible API which may or may not expose either all features of the platform or all features of the OpenAI API), and the differences are not cosmetic, but since LiteLLM itself just presents an OpenAI-compatible API, it can't be providing acccess to other vendor features that don't map cleanly to that API, and I don't think its likely to be using the native API for each and being more complete in its OpenAI-compatible implementation of even the features that map naturally than the first-party OpenAI-compatibility APIs.)

by circularfoyers1 days ago|

prev|

[-]

Comparing this project to is-odd seems very disingenuous to me. My understanding is this was the only way you could use llama.cpp with Claude Code for example, since llama.cpp doesn't support the Anthropic compatible endpoint and doing so yourself isn't anywhere near as trivial as your comparison. Happy to be corrected if I'm wrong.

by jerieljan1 days ago|

parent|

[-]

That's a correct example, and I agree, it is disingenuous to just trivially call this an `is-odd` project.

Back in the days of GPT-3.5, LiteLLM was one of the projects that helped provide a reliable adapter for projects to communicate across AI labs' APIs and when things drifted ever so slightly despite being an "OpenAI-compatible API", LiteLLM made it much easier for developers to use it rather than reinventing and debugging such nuances.

Nowadays, that gateway of theirs isn't also just a funnel for centralizing API calls but it also serves other purposes, like putting guardrails consistently across all connections, tracking key spend on tokens, dispensing keys without having to do so on the main platforms, etc.

There's also more to just LiteLLM being an inference gateway too, it's also a package used by other projects. If you had a project that needed to support multiple endpoints as fallback, there's a chance LiteLLM's empowering that.

Hence, supply chain attack. The GitHub issue literally has mentions all over other projects because they're urged to pin to safe versions since they rely on it.

by 1 days ago|

prev|

[-]

deleted