Is this true for coding tasks? Genuinely asking because I don't yet have the hardware to be able to run/compare. I thought I remembered the sentiment that the 27B dense was better than the 35B MoE

The 3 billion includes all active experts, and then other shared weights, attention components, embeddings.

In the qwen 3.5 35b a3b series, each expert is ~3 million parameters. There are 40 layers which each have 1 shared expert and 8 routed experts. Roughly 1.1B of the 3B active are experts.

Last year, it would have been impossible to run a 35B A3B model on old Pascual gpu and make it somehow usable. But for some reason now it just works even with a better finetuned version. So clearly there is more to it that just having it theoretically fit the system. Look for example how Strata works on certain models and systems compared to its llamacpp counterpart.

Use Vulkan build of llama.cpp to run low parameter MoE models in your IGPU and CUDA llama.cpp to run dense models in your DGPU.

Ofcourse you need to set your expectations right on what problem you want to solve, I've not been happy with using quantized models for coding even on a decent laptop; where as for NLP, ASR, STT etc. work great with this setup.

I might write more about this setup soon.

I believe in your case the model quality matters a lot. To my (trust me bro) experience any MOE model, finetuned or not, with params < 19B even with Q4-quant always give lackluster results. So any Qwen2.5 wouldn't suffice to even simple coding tasks. For your case maybe you can bump it up to a Gemma4 E2B/E4B or Qwen3.5-9B. Also make sure that your current build of llamacpp actually uses your GPU with CUDA.

You'd have better luck with pi instead of OpenCode, and see how far you can get that context window higher. For any halfway actual work you need bare minimum 64k, ideally 128k+. You can use 8bit kv cache, despite whatever folklore you may hear otherwise.

If you drop the thinking level to low or even off the context will go further at the cost of the models intelligence. That might also help you get some answer but since such a question will read so many files into that context its the wrong approach. This is one of the bigger limitations of local LLMs, you can't do work that spans across a lot of files because you are almost always context limited. There is a limit of 1M even with the frontier so even that can't review an entire project. Focussing in on individual files or small groups is more achievable.

So just to be specific the -c needs to be >65536 and depending on the size of what is in the folder might very well not be enough.

Not sure what llamacpp build you are using but on mine using this fork: https://github.com/TheTom/llama-cpp-turboquant on commit 407f3237bfb3eeaff61546797de3d8c1a96be748 compiled for Pascal, I am able to push my current system to 131k context with minimal loss in quality. For reference this is my config.ini file for this model (Cyber is the better uncensored version btw):

[*] # Global defaults jinja = 1 metrics = 1 threads = 12 threads-batch = 12 flash-attn = on sleep-idle-seconds = 99999 timeout = 99999 no-mmap = 1 mlock = 1 ctk = turbo4 ctv = turbo2

[qwen36-a3b] model = /root/llm/models/Cyber-Tiel-Coder-35B-A3B-MTP-UD-IQ4_XS.gguf alias = qwen36-a3b n-cpu-moe = 32 temp = 0.6 top-p = 0.95 top-k = 20 min-p = 0.0 presence-penalty = 0.0 repeat-penalty = 1.0 reasoning = on n-predict = 32768 reasoning-format = deepseek reasoning-budget = 16384 reasoning-budget-message = "Proceed to final answer" reasoning-preserve = true swa-full = true np = 2 batch-size = 4096 ubatch-size = 2048 cache-reuse = 256 ngl = auto ctx-checkpoints = 33 ctx-size = 262144 no-mmproj = true no-mmproj-offload = true spec-type = draft-mtp spec-draft-n-max = 2

I am also certain that these flags could be improved somehow depending on your system.

I would go ahead and give it at least 64k but the more the better really!

I run the kv cache at q8 for keys and q4 for values, but going below q8 is generally not recommended.