(llmspeed.dev)
Currently on my 9 year old laptop with a GTX 1070 Max-Q 8GB vram, 16 gb ddr4 ram and i7 8750h. I am running a finetuned Qwen 3.5-35B-A3B (Tiel-Coder-35B-A3B-GGUF-MTP) with 4-bit quant on a llamacpp fork (Turboquant) with 131k context at around 16-19 tk/s and 350-500 tk/s prefil using Pi harness. It is amazing for writing scripts and doing medium coding tasks and summarizing text for me. After around 100k context it goes down to 9 tk/s but that is to be expected.
If you ever ask any frontier LLM or these types of benchmarks, they would probably say that the best model I can run on my system is a crappy 9B model or something, which is just completely false. So I really wonder how much more you can pull out from more powerful systems and much better open weight models.
RAM is going to be the big bottleneck for MOE (at least for now) as it will determine the quantisations you could run.
On a 3080 10GB and 32GB of RAM I can run the qwen-next coder model and that is a 127B model and it gets 40-50 Token/s with 65k context. That is a lot more usable than a Quantised Qwen3.8-27B in my experience both because its faster and more capable.
I am looking forward to Qwen4 soon, I am hoping we get some more 27-35B MOE models, those are going to be really good for local hardware.
Is this true for coding tasks? Genuinely asking because I don't yet have the hardware to be able to run/compare. I thought I remembered the sentiment that the 27B dense was better than the 35B MoE
It's all about trade-offs: how much of the model you want to load into VRAM; how much VRAM you want to reserve for context; how much quantization you/the model can tolerate; how many tokens/second you can tolerate; whether you want to use the VRAM for anything else (graphics, Text-to-Speech, etc.); etc.
The 3 billion includes all active experts, and then other shared weights, attention components, embeddings.
In the qwen 3.5 35b a3b series, each expert is ~3 million parameters. There are 40 layers which each have 1 shared expert and 8 routed experts. Roughly 1.1B of the 3B active are experts.
Last year, it would have been impossible to run a 35B A3B model on old Pascual gpu and make it somehow usable. But for some reason now it just works even with a better finetuned version. So clearly there is more to it that just having it theoretically fit the system. Look for example how Strata works on certain models and systems compared to its llamacpp counterpart.
I'm personally sitting on an extremely limited Geforce GTX 1660 6GB, and would like to see how far one could go even with such a crappy card.
At least it was quite amazing to see the difference between CPU-based and even this crappy GPU-based inference when trying out some tiny models in jan.ai. It's still like orders of magnitude faster at least.
Use Vulkan build of llama.cpp to run low parameter MoE models in your IGPU and CUDA llama.cpp to run dense models in your DGPU.
Ofcourse you need to set your expectations right on what problem you want to solve, I've not been happy with using quantized models for coding even on a decent laptop; where as for NLP, ASR, STT etc. work great with this setup.
I might write more about this setup soon.
I tried llama.cpp with offloading "all what could be" to my GPU (GTX 750 Ti 2 GB). I could run local Qwen2.5-Coder quite fast (I do not remember but something like 25-30 t/s). However this model generates bad code (I tried Python and JavaScript). So, for my setup it is better to chat with free ChatGPT and do copying-pasting between web chat and code editor.
Is it even worth for trying something on my GPU? What do you think?
And yes, I use Claude Code (Opus 5.5) and Codex in my job but for personal stuff I would prefer trying free solutions (it is the reason I tried local LLM). I do not want to buy subs, at leats for now.
I believe in your case the model quality matters a lot. To my (trust me bro) experience any MOE model, finetuned or not, with params < 19B even with Q4-quant always give lackluster results. So any Qwen2.5 wouldn't suffice to even simple coding tasks. For your case maybe you can bump it up to a Gemma4 E2B/E4B or Qwen3.5-9B. Also make sure that your current build of llamacpp actually uses your GPU with CUDA.
I am running on a Ryzen 5950X, 64GB DDR4 Ram and a Radeon RX 6700 XT with 12GB... I don't expect wonders, but I have between 11 and 120 (!) t/s.
I told it an open ended question: "What would you improve in this file at {location} in {bigger project}" and the context window seems to be too small. It's running in circles.
I must be doing something wrong.
Again, I am not expecting wonders and I am totally willing to accept that "I am holding it wrong".
llama-server \ -m ~/models/tiel-coder/Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf \ -ngl 20 \ -c 16384 \ --jinja \ --alias tiel-coder
using it in opencode
{ "$schema": "https://opencode.ai/config.json", "provider": { "llamacpp": { "npm": "@ai-sdk/openai-compatible", "name": "Local llama.cpp", "options": { "baseURL": "http://127.0.0.1:8080/v1" }, "models": { "tiel-coder": { "name": "Tiel-Coder 35B A3B", "limit": { "context": 16384, "output": 4096 } } } } }, "model": "llamacpp/tiel-coder" }
You'd have better luck with pi instead of OpenCode, and see how far you can get that context window higher. For any halfway actual work you need bare minimum 64k, ideally 128k+. You can use 8bit kv cache, despite whatever folklore you may hear otherwise.
If you drop the thinking level to low or even off the context will go further at the cost of the models intelligence. That might also help you get some answer but since such a question will read so many files into that context its the wrong approach. This is one of the bigger limitations of local LLMs, you can't do work that spans across a lot of files because you are almost always context limited. There is a limit of 1M even with the frontier so even that can't review an entire project. Focussing in on individual files or small groups is more achievable.
So just to be specific the -c needs to be >65536 and depending on the size of what is in the folder might very well not be enough.
Not sure what llamacpp build you are using but on mine using this fork: https://github.com/TheTom/llama-cpp-turboquant on commit 407f3237bfb3eeaff61546797de3d8c1a96be748 compiled for Pascal, I am able to push my current system to 131k context with minimal loss in quality. For reference this is my config.ini file for this model (Cyber is the better uncensored version btw):
[*] # Global defaults jinja = 1 metrics = 1 threads = 12 threads-batch = 12 flash-attn = on sleep-idle-seconds = 99999 timeout = 99999 no-mmap = 1 mlock = 1 ctk = turbo4 ctv = turbo2
[qwen36-a3b] model = /root/llm/models/Cyber-Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf alias = qwen36-a3b n-cpu-moe = 32 temp = 0.6 top-p = 0.95 top-k = 20 min-p = 0.0 presence-penalty = 0.0 repeat-penalty = 1.0 reasoning = on n-predict = 32768 reasoning-format = deepseek reasoning-budget = 16384 reasoning-budget-message = "Proceed to final answer" reasoning-preserve = true swa-full = true np = 2 batch-size = 4096 ubatch-size = 2048 cache-reuse = 256 ngl = auto ctx-checkpoints = 33 ctx-size = 262144 no-mmproj = true no-mmproj-offload = true spec-type = draft-mtp spec-draft-n-max = 2
I am also certain that these flags could be improved somehow depending on your system.
I would go ahead and give it at least 64k but the more the better really!
I run the kv cache at q8 for keys and q4 for values, but going below q8 is generally not recommended.
would drop it from fp16 to 8 bits. You don't want to go below 8 bits if you can possibly avoid it and if you really do need to squeeze just drop the v value to q4_0 but leave the key alone. Unlike model quantisation this really makes things worse quickly.
There are many knobs to tweak to make it faster, which the site ignores.
It's hard to know if we're at the beginning or the end of this trend. If manufacturing continues to focus on high end datacenter gpus at $40k a pop, and we continue to have supply constraints, this could last a while.
On the other hand, if Chinese models continue to nip at the tails of the frontier, cheaper by 20x, running on new Chinese chips, the whole bottom could fall out of the US market, and things could get very weird.
in my use case, the model might need to analyze a given task first then generate and execute a database query and finally send the results back to the client. to what extent could your setup handle this kind of workflow and how accurate are the results?
good job.
1. It is difficult to figure out if dropping $10k+ on hardware will make a meaningful difference for running local models.
2. Downloading each model and each quantization of that model to see the speed seems rude to free services like hugging face, so now I can see things like "with my hardware, going to a Q8 only results in 3 more tokens per second"
> This combination won't run. Qwen 3.6/3.8 27B at FP16 needs 54.6 GB — the AMD RX 9070 XT 16GB has 14.9 GB usable. No quantization of this model fits; pick a smaller model or more memory.
While it is true that it won't fit in your GPU memory, it demonstrably does run on my hardware. Llama.cpp supports putting some layers on the GPU and running others on the CPU.
Also, that message says "No quantization of this model fits" but switching to a different quantization of that model (for example, Q8_0) makes the message go away?
Time to first token (TTFT) on 5,687 tokens, like pasting a short code block into the prompt, takes 36 seconds vs 4.7 seconds. It also breaks down the seconds for each step like KV and prefilling.
Gemma4 31B TTFT on the M5 Max 128gb with same setup is showing as 19 seconds.
Depending on who you sign up for, that's within the introductory credits (usually like 5-10$) so it's actually $0.
No clue about details, but Sebastian Raschka's site reports its kv cache usage at 210kB per token, which is like 6x more than this site claims ("0.273 GB per 8k"). Also, the site claims qwen 27b to use 2x more memory per token than gemma, which is the opposite from what I know.
- multi GPU configuration (hetero and homo)
- PCIe gen chooser (3.0/4.0/5.0)
- is nvlink enabled (2 cards, 2 x 2 cards, 3 x 2 cards)
- is PCIe bifurcated or CPU native (threadripper pro for example)
It won't win benchmarks of course, but 20 tps generation is okay and letting it chug away on my local kubernetes cluster to troubleshoot is pretty nice. First start is pretty slow though, parsing is around 100-200 tps. Which is crap for long questions or harnesses like opencode which start around with thousands of tokens. Once cache is warm, pretty enjoyable.
I think it could have way more use cases where it's just 'good enough'. Email categorization, voice commands for the home. 1 year ago this was still a very miserable experience, but qwen 9B + better cache quantization and more tricks in the latest llm engines makes it viable.
Will it do difficult architectural refactors by just giving a one-shot prompt like opus allows, nope, not gonna happen.
Yeah simple, but menial refactors where IDEs don't have clear cut refactors yet. Or languages like CUE which has awesome tooling, but not so great refactors in its lsp yet.
I don't think in general people care about long list of models, they just care about the best one.
Also dual gpu is common setup, there should be ie. 2x 4090.
Asus ProArt P14, Surface Ultra, Yoga Pro 9n, etc.
I am hoping the cheaper N1 models will also come out soon, but might be several months.
Also would be good to see a few more quant and model options.
Unless of course he's driving some weird lambo that dumps 200k of context into their chats. Who knows, I feel like we're on a yellow submarine.
I wished there was such a tool before picking my hardware, it's useful. But the estimates needs to be somewhat correct.
It would be neat to also include NVFP4 variants for the Blackwell hardware.