upvote

points

by Maxious16 hours ago |
comments
upvoteby danielhanchen14 hours ago|
next
[-]
Oh I didn't expect this to be on HN haha - but yes for our new benchmarks for Qwen3.5, we devised a slightly different approach for quantization which we plan to roll out to all new models from now on!
reply
upvoteby nnx13 hours ago|
parent|
next
[-]
Can you describe what is this slightly different approach and why it should work on all models?
reply
upvoteby hedora7 hours ago|
parent|
prev|
[-]
Nice! Your stuff ran LLMs extremely well on < $500 boxes (24-32GB ram) with iGPUS before this update.

I’m eager to try it out, especially if 16GB is viable now.

reply
upvoteby Kayou16 hours ago|
prev|
next
[-]
Wait, the Q4 quantization which is more than 20GB fits in your 16GB GPU ? I didn't know that was possible, I was always restricting myself to smaller model than the VRAM I had
reply
upvoteby Maxious15 hours ago|
parent|
next
[-]
Yep. These Mixture of Experts models are well suited for paging in only the relevant data for a certain task https://huggingface.co/blog/moe

There's some experiments of just removing or merging experts post training to shrink models even more https://bknyaz.github.io/blog/2026/moe/

reply
upvoteby vlovich1238 hours ago|
parent|
next
[-]
MoE is not suited for paging because it’s essentially a random expert per token. It only improves throughput because you reduce the memory bandwidth requirements for generating a token since 1/n of the weights are accessed per token (but a different 1/n on each loop).

Now shrinking them sure, but I’ve seen nothing that indicates you can just page weights in and out without cratering your performance like you would with a non MoE model

reply
upvoteby FuckButtons7 hours ago|
parent|
[-]
Not entirely true, it’s random access within the relevant subset of experts and since concepts are clustered you actually have a much higher probability of repeatedly accessing the same subset of experts more frequently.
reply
upvoteby vlovich12357 minutes ago|
parent|
[-]
It’s called mixture of experts but it’s not that concepts map cleanly or even roughly to different experts. Otherwise you wouldn’t get a new expert on every token. You have to remember these were designed to improve throughput in cloud deployments where different GPUs load an expert. There you precisely want each expert to handle randomly to improve your GPU utilization rate. I have not heard anyone training local MoE models to aid sharding.
reply
upvoteby cagenut14 minutes ago|
parent|
[-]
is there anywhere good to read/follow to get operational clarity on this stuff?

my current system of looking for 1 in 1000 posts on HN or 1 in 100 on r/locallama is tedious.

reply
upvoteby bee_rider10 hours ago|
parent|
prev|
[-]
That blog post was super interesting. It is neat that he can select experts and control the routing in the model—not having played with the models in detail, tended to assume the “mixing” in mixture of experts was more like a blender, haha. The models are still quite lumpy I guess!
reply
upvoteby segmondy16 hours ago|
parent|
prev|
next
[-]
llama.cpp is designed for partial offloading, the most important part of the model will be loaded into the GPU and the rest on system ram. I run 500B+ models such as DeepSeek/KimiK2.5/GLM-5 without having that much GPU vram.
reply
upvoteby Koffiepoeder14 hours ago|
parent|
prev|
next
[-]
The A3B part in the name stands for `Active 3B`, so for the inference jobs a core 3B is used in conjunction with another subpart of the model, based on the task (MoE, mixture of experts). If you use these models mostly for related/similar tasks, that means you can make do with a lot less than the 35B params in active RAM. These models are therefore also sometimes called sparse models.
reply
upvoteby nurettin14 hours ago|
parent|
prev|
[-]
This is why they say "A3B" meaning only 3B is active at a time, limiting VRAM usage.
reply
upvoteby roxolotl13 hours ago|
prev|
next
[-]
What method are you using to do that? I’ve been playing with llama.cpp a lot lately and trying to figure out the cleanest options for getting a solid context window on 32gb vram and 64gb system ram.
reply
upvoteby jychang13 hours ago|
parent|
[-]
32GB vram is more than enough for Qwen 3.5 35b

You can just load the Q4_K_XL model like normal, and put all tensors on GPU without any -ot or --cpu-moe flags.

If you need a massive context for some reason where model+kv cache won't fit in 32gb, then use -ot to move the ffn moe experts for 1-2 layers into RAM. You'll get a speed hit (due to loading params from slower RAM instead of fast VRAM) but it'll work.

reply
upvoteby roxolotl13 hours ago|
parent|
[-]
Nice ok I’ll play with that. I’m mostly just learning what’s possible. Qwen 3.5 35b has been great without any customizations but it’s interesting to learn what the options are.
reply
upvoteby mirekrusin15 hours ago|
prev|
next
[-]
2x RTX 4090, Q8, 256k context, 110 t/s
reply
upvoteby instagib8 hours ago|
parent|
[-]
1 4090, Qwen3.5-35B-A3B-UD-MXFP4_MOE, 64k context, 122 t/s. Llama.cpp
reply
upvoteby cpburns200912 hours ago|
prev|
next
[-]
Does llama.cpp support Qwen3.5 yet? When I tried it before, it failed saying "qwen35moe" is an unsupported architecture.
reply
upvoteby hnfong11 hours ago|
parent|
next
[-]
Yes, but make sure you grab the latest llama.cpp release

New model archs usually involve code changes.

reply
upvoteby cpburns200910 hours ago|
parent|
[-]
Awesome! It looks like the llama.cpp-hip AUR was updated today to b8179, and it works.
reply
upvoteby reactordev12 hours ago|
parent|
prev|
[-]
You would need the Dynamic 2.0 GGUF as discussed in the article.

But mmmmmm, Q8_K_XL looks mighty nice.

reply
upvoteby RS-23214 hours ago|
prev|
next
[-]
That’s intriguing. I have the same card, maybe I should give it a go. Curious about your CPU/RAM/storage capacity as well.

Any resources for configuring the local setup?

My entire home media stack is a single compose file in a WSL distro so it would be cool if local LLM worked the same way.

reply
upvoteby jychang16 hours ago|
prev|
[-]
Not really breakthroughs, more like bugfixes for their broken first batch.
reply
upvoteby danielhanchen14 hours ago|
parent|
[-]
No this is false - unsure if you saw our new blog - https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks which shows SOTA on nearly all bits, and we shared all our research as well
reply
upvoteby zargon8 hours ago|
parent|
next
[-]
Explain what about that statement is false. Your original Q4_K_XL quant was broken. People noticing that it was a total outlier among other quants is what prompted this "research". Your own data proves that your new release fixes the bugs of your original, in order to match AesSedai's PPL. Fixing bugs is great. Searching for the best quant mix is helpful. I use your quants and appreciate your work. But whitewashing this situation dilutes trust and good will.
reply
upvoteby jychang14 hours ago|
parent|
prev|
[-]
Yeah, I saw that yesterday. The blog post does not explain why/how the Qwen 3.5 quants uploaded on 2/27 are different from the files uploaded on 2/24.

Old 2/24 Q4_K_XL commit (pre bugfix files): https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/commit/7...

Questions for a postmortem that the blog post left unanswered:

- Why the change? Is it just to improve PPL/KLD? Sure, we can assume PPL and KLD are not perfect benchmarks. If yes, then why change the quantization anyways? Or was the old 2/24 quant actually much worse performing in the real world?I presume the Q4_K_XL quant using mxfp4 was the issue? If the 2/24 files having a lower PPL is an actual issue due to low quality tensors, then why not just say that?

- What were the main tensors that had the quantizations changed from 2/24 to 2/27? Did you now quantize attention tensors differently? Or perhaps ssm? T

- What was it changed from? Was it changed from mxfp4 or q4_k to q8, or something else?

A quick sentence in the blog post saying "ok, we've confirmed that using mxfp4 (or q3 or whatever) in the attention/ssm/biases/norms/etc is a bad idea, we had that in our old models on 2/24 and our new models today are better" that would make it clear. As it's written, it's trying to both say "PPL/KLD don't actually reflect real world quality" and "we changed our quant to increase PPL/KLD" at the same time, which seems contradictory.

reply