upvote

points

by mark_l_watson20 hours ago |
comments
upvoteby cube0017 hours ago|
next
[-]
Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385
reply
upvoteby LeBit15 hours ago|
parent|
next
[-]
When that article was first published , I started looking into llama.cpp. With the help of an LLM I researched the knobs to turn that made most sense.

Things went from "local models are useless unless you have a 512GB GPU I guess" to "oh.. I can do a lot of stuff locally now!"

reply
upvoteby chorizo13 hours ago|
parent|
[-]
Had a similar experience. Llama.cpp compiled natively; parameter sweep to find best options fitting my use case for the qwen models with 16GB VRAM. The whole thing packaged into a portable container.
reply
upvoteby mark_l_watson1 hours ago|
parent|
prev|
next
[-]
From the Ollama docs for this new model:

Ollama's MLX engine provides state-of-the-art performance on Apple Silicon, with support for DFlash and image input:

ollama run muse-glimmer:30b-mlx

reply
upvoteby lenerdenator17 hours ago|
parent|
prev|
[-]
What do you use instead?
reply
upvoteby SwellJoe12 hours ago|
parent|
next
[-]
If you need a GUI, Unsloth Studio and LM Studio are both great frontends for llama.cpp. If you don't need a GUI, llama.cpp is the business for single-user deployments. Easy to use, always gets new model support very quickly, built-in Hugging Face client/cache support, works on probably everything (Mac, ROCm, CUDA, Vulkan, etc.).
reply
upvoteby computershit17 hours ago|
parent|
prev|
next
[-]
I use llama.cpp w/ llama-swap

https://github.com/ggml-org/llama.cpp https://github.com/mostlygeek/llama-swap

reply
upvoteby rancor15 hours ago|
parent|
next
[-]
FYI, llama-server can now be run in router mode so llama-swap is probably only needed for more exotic scenarios.
reply
upvoteby computershit11 hours ago|
parent|
next
[-]
I’m using llama-swap because it can manage arbitrary backends, not just llama-server instances. I have llama.cpp chat and embedding models running alongside whisper-server all behind a single endpoint with per-model TTLs so they don't fight over the limited vram I have available on this box. Native routing could replace the llama.cpp part but not whisper so I guess I'm exotic ;)
reply
upvoteby kingo5513 hours ago|
parent|
prev|
[-]
I'm running it in router mode, but people on Reddit were recommending people use llama-swap instead. Am I missing something by using router mode?
reply
upvoteby mudkipdev11 hours ago|
parent|
[-]
I believe it's useful for running multiple llama.cpp forks at the same time (e.g. a model you want requires special patching)
reply
upvoteby LeBit15 hours ago|
parent|
prev|
[-]
Thanks for the link to llama-swap. Didn’t know about it and will definitely install it.
reply
upvoteby reilly300017 hours ago|
parent|
prev|
next
[-]
try oMLX or vMLX - both great projects that offer some amazing performance optimizations for Apple Silicon that utilize UMA and NVME caching efficiently.

https://omlx.ai https://vmlx.net

That said, it's been a few weeks since I've looked so maybe llama.cpp has those features now... they really do move that quickly.

reply
upvoteby scotty7911 hours ago|
parent|
prev|
next
[-]
Jan is so much better.
reply
upvoteby soupspaces16 hours ago|
parent|
prev|
next
[-]
The link in that discussion has a section about that
reply
upvoteby nickthegreek15 hours ago|
parent|
prev|
[-]
lm studio
reply
upvoteby khimaros20 hours ago|
prev|
next
[-]
seems to underperform on Terminal Bench compared with qwen3.6-27b: 51.7 vs 60.7
reply
upvoteby mark_l_watson19 hours ago|
parent|
[-]
To be honest, I never give benchmarks a look. I just use the models for whatever I need to work on, so I can't really make comparisons that are useful for other people.
reply
upvoteby 17 hours ago|
prev|
next
[-]
deleted
reply
upvoteby spaceywilly20 hours ago|
prev|
[-]
Newb question but I’m curious what would help it to run faster? Would it need more vRAM or just system memory?
reply
upvoteby spmurrayzzz20 hours ago|
parent|
next
[-]
The biggest gain you'll get is faster memory, provided you have enough capacity to load all the weight into vram. The DGX sparks and Apple silicon memory bandwidth (and also memory access latency) drag down the decode speed quite a bit.

I have two GPU rigs both with 2x RTX Pro 6000, can get ~250 tk/s decode with deepseek-v4-flash in native mixed precision. For context, in antirez's dwarfstar project he only gets ~20-40 tk/s on the same model @ 2bpw on M5 Max.

The latter is for sure usable if it's your only option, but it's really hard for me to personally go back to speeds like that when I've experienced the former.

(Also worth noting dwarfstar only has experimental support for dspark spec dec, when that lands it will definitely give a big boost at higher acceptance rates)

reply
upvoteby wincy19 hours ago|
parent|
prev|
[-]
It runs very quickly on my RTX 5090 fwiw. Whole thing is loading entirely into vRAM with a ~130k context size (the max) fitting as well.
reply
upvoteby codazoda18 hours ago|
parent|
[-]
That's a $5k 32GB card for anyone who doesn't know all these off the top of their heads (like myself).
reply