upvote
Caveat: this kind-of-works. There are plenty of subtle bugs around this. For example, it's hard to have a default model, and clients need to select the model explicitly in every API request, which makes it difficult to work with several models and multiple clients.
reply
Is that with vanilla llama.cpp or with the third-party llama-swap manager? Last time I checked llama-swap was still the go-to solution, although I admit I haven't looked into it further.
reply
llama.cpp has router mode these days, swapping out models for you, you no longer need llama-swap.
reply
I use it on the daily. Haven't exactly missed ollama.
reply
If you revisit my comment and pay attention to the opener:

> but you might not be aware that llama-server can do multi-model for a while now

you will see that the sentence structure clearly implies both a change compared with a prior state and also lack of any third-party thing.

So the answer to the question has already been encoded as text available.

_

I can see the desire for explicit validation though. For that, I would propose a sentence structure like

> Oh cool! That means that llama-swap is now superseded/no longer needed?

That shows that you've read and understand the message, gives you the double-check and might on top spark a conversation about how these solutions compare. Plus that if the guy you're commenting too has spoken nonsense, they need to backpedal.

reply
btw, llama-swap provides a nice UI for monitoring performance and logs, and even the ability to stop an infinite session that consumes GPU resources (sometimes that happens).

Does the llama.cpp UI provide the same? If not, it is too early to say that llama-swap is “superseded/no longer needed.”

reply
It's also convenient if you want to run separate builds behind a single config, sometimes there's a PR I want to run before it's merged
reply
> (Optimized by you through testing. Not that AI)

Why not optimized by AI through testing ? Give it a test set to work on and let it loose.

reply
AI doesn't necessarily know what feels like a good tradeoff to you. I'm sure it could help guide you though.
reply
This is all quantifiable. I regularly have my model run benchmarks against all the config permutations and then choose the best based on my criteria, which typically boil down to trading prefill and decode times
reply
^ This.

Intent is the answer and AI has none.

reply
deleted
reply
[dead]
reply