upvote

points

by akie14 hours ago |
comments
upvoteby madduci14 hours ago|
next
[-]
Everything lower than 4/5 is really bad
reply
upvoteby latentsea12 hours ago|
parent|
next
[-]
This mentality is really pervasive and yet I've found when a lower than 4-bit quant of a given model was all I could use, then using that was often better than using a 4-bit quant of a model that I could. YMMV. It just really made me rethink my stance on the whole thing. Most people likely see something like 2-bit and just feel like the want to puke without ever actually learning whether or not it can handle their tasks and how well relative to other models they run. I've been running Qwen3.8-Flash-Next at Q4, but was crazy surprised at the performance of Unsloth's IQ1_M of DeepSeek v4.1 Flash.
reply
upvoteby sheepscreek3 hours ago|
parent|
next
[-]
Thanks for sharing this - I would not have looked into it otherwise. Apparently model retention is ~98% of the original at IQ1_M if I read that right. Crazy. Excited to try this now.
reply
upvoteby ferroman8 hours ago|
parent|
prev|
next
[-]
Qwen3.8-Flash-Next at Q2 is often looping itself with repetitions.
reply
upvoteby pyrolistical10 hours ago|
parent|
prev|
[-]
You don’t know what you don’t know
reply
upvoteby tredre37 hours ago|
parent|
[-]
It works the other way too! People don't know that 3 or 2 bits can still be good because they've read the hivemind's opinion that <4bit is unusable trash and they've never experimented themselves. The reality is a lot more nuanced than that, especially for models as large as 284B.
reply
upvoteby smallerize14 hours ago|
parent|
prev|
next
[-]
Well it's bad compared to the original model, but is it bad compared to what else you could run in 64 GB?
reply
upvoteby serf14 hours ago|
parent|
next
[-]
probably; 32-64 is the range where local models get extraordinarily useful, and in my experience any sub-4 quants get ridiculously incoherent, which is why those models tend to try to 'wow' people with tok/s rather than benchmarks.

haven't tried it yet though.

reply
upvoteby cyanydeez12 hours ago|
parent|
[-]
Qwen3.8-Flash-Next fits in 80GB on strix halo. That's when I started real "vibe" coding with them. Speed is still an issue if I want to move fast and break things, but I've built a little universe for development (I assume many people have also) and it's cruising.

But yeah, 32-64 is the gateway I think. If Qwen4 can make a leap in the flast-next MoE and hit 64 GB on VRAM w/context, that's definitely the cloud killer to me.

reply
upvoteby Tepix14 hours ago|
parent|
prev|
next
[-]
I’m wondering myself: Do bigger models suffer less from quantization? Are there papers?
reply
upvoteby ricardobeat13 hours ago|
parent|
[-]
They seem to degrade a lot less.

You don't need papers, good quants are usually published with benchmark data, for example: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-... which maintains parity with FP8 even at 2 bit.

reply
upvoteby alfiedotwtf12 hours ago|
parent|
[-]
Always check the label and if pain persists, see your doctor!

The ISTA accuracy looks good on that page, but it's only a chosen handful. Instead, look at most of the graphs on Unsloth's Qwen 3.8 Flash Next - https://unsloth.ai/docs/models/qwen3.8-next

As always, check the quant vs KLD graphs (half way down that page).. I found that not all quants are the same - Unsloth's GLM Q4 -> Q3 degrades a lot faster than their Minimax M3.1 Q4 -> Q3 ggufs.

reply
upvoteby pitchlatte13 hours ago|
parent|
prev|
[-]
you can get a lot done with models well under that - the google and qwen 20-30B class models are fantastic
reply
upvoteby rpdillon11 hours ago|
parent|
prev|
next
[-]
Deepseek v4 flash 0731 in a 2-bit quant has been useful for me. Other models don't seem to hold up nearly as well at 2-bit. Most of my other models are 5-6 bit quants because of this.
reply
upvoteby LeBit13 hours ago|
parent|
prev|
next
[-]
This but also the kv quantization shouldn’t be lower than 8 bits.
reply
upvoteby mt_13 hours ago|
parent|
prev|
next
[-]
Next frontier is achieving good output with that kind of quant dimension
reply
upvoteby ranger_danger12 hours ago|
parent|
prev|
[-]
There are solutions for that: https://prismml.com/news/bonsai-2-27b

(and before you crap on it, this is not the same as the original Bonsai, this newer one is much better, and upstream support in llama.cpp etc. is being worked on)

reply
upvoteby adrian1713 hours ago|
prev|
next
[-]
I’m confused too and the slop readme doesn’t help at all.

> Shrinks DeepSeek-V4-Flash from 76.87 GiB (which crashes in OOM on 64 GB Linux) down to 56.56 GiB

And later

> Standard IQ2_XXS (76.87 GiB, Unpatched)

Except unsloth’s IQ2_XXS GGUF is 91GB, even their smallest IQ1_S is 82.5GB. So I wonder which model/quant the number even refers to.

Also the mentions of “compression” and “MoE pruning” also make it sound lobotomized even more than unsloth’s IQ1 quants.

reply
upvoteby adrian179 hours ago|
parent|
[-]
PS: only now noticed that 76.87GiB is 82.5GB, which happens to be size of the unsloth IQ1_S. The “technical” paper also mentions IQ1_S at some point. Sounds to me like the author’s model or the author mixed IQ2_XXS with IQ1_S at some point.

Also, just noticed that “moe pruning” likely means “the model is supposed to run 6(DS)/8(Qwen) experts per token per layer, but we only run 4 to save on time”.

Either way, the final gguf is 75% of the size of unsloth IQ1_S, which by itself should tell you a lot about its quality; they (somehow; I have no interest digging that info from that slop paper) shrink the (already IQ1) FFN weights, and shrink some attention weights from Q5 to Q2. I’m surprised it was even able to generate the text in the demo.

reply
upvoteby Tepix14 hours ago|
prev|
[-]
This 100%

Reddit is flooded with these breathless headlines “xyz tokens/s on my light switch”. It’s always quantizations below 4 bits per weight. Mostly useless.

reply