People who use Ollama generally (not everyone obviously) don't always clearly understand what quantization they use when running models, so people end up saying "I tried running Qwen 3.8 27b locally and it was dumb" while Ollama would default to a Q4 version of the model, which has very different results from the BF16 weights, doesn't really speak to the model itself because it's been so quantized in that case.

Sure, makes things easier, but tons of people misunderstand what they're using, then base and share their experiences on that, without really specifying what exact weights they use too.

For a single local user, using llama.cpp directly shouldn't be a problem if you're already using Ollama's CLI, it works basically the same except you manage weights yourself, and if you put your favorite agent to make sense of the faux "registry + image layers" Ollama has prepared locally for you, you can reuse the files you've already downloaded with Ollama.

I know nothing about "vLLM" -> anything better compared to "llama.cpp"?

I started with "Ollama" (precompiled version) and it worked and was good enough to understand the very basics.

Then I downloaded the sourcecode of "llama.cpp", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck with "llama.cpp" (and "ik_llama.cpp" in very few cases).

I honestly don't know what made "Ollama" (precompiled) so much slower than "llama.cpp" (compiled locally) at that time and I'm too lazy to doublecheck now, in any case I now absolutely love all the knobs that "llama.cpp" has to tune your hardware setup & your workload, which is the reason why I recommend it.

From what I've heard, Ollama has a bad reputation because it's a thin wrapper around llama.cpp without attributing it properly, thereby stealing recognition from the maintainers doing most of the work

> is there something fundamentally wrong with Ollama ?

Yes https://news.ycombinator.com/item?id=47788385