undefined

points

[-]

This mentality is really pervasive and yet I've found when a lower than 4-bit quant of a given model was all I could use, then using that was often better than using a 4-bit quant of a model that I could. YMMV. It just really made me rethink my stance on the whole thing. Most people likely see something like 2-bit and just feel like the want to puke without ever actually learning whether or not it can handle their tasks and how well relative to other models they run. I've been running Qwen3.8-Flash-Next at Q4, but was crazy surprised at the performance of Unsloth's IQ1_M of DeepSeek v4.1 Flash.

by sheepscreek3 hours ago|

parent|

[-]

Thanks for sharing this - I would not have looked into it otherwise. Apparently model retention is ~98% of the original at IQ1_M if I read that right. Crazy. Excited to try this now.

by ferroman8 hours ago|

parent|

prev|

[-]

Qwen3.8-Flash-Next at Q2 is often looping itself with repetitions.

by pyrolistical10 hours ago|

parent|

prev|

[-]

You don’t know what you don’t know

by tredre37 hours ago|

parent|

[-]

It works the other way too! People don't know that 3 or 2 bits can still be good because they've read the hivemind's opinion that <4bit is unusable trash and they've never experimented themselves. The reality is a lot more nuanced than that, especially for models as large as 284B.

by smallerize14 hours ago|

prev|

[-]

Well it's bad compared to the original model, but is it bad compared to what else you could run in 64 GB?

by serf14 hours ago|

parent|

[-]

probably; 32-64 is the range where local models get extraordinarily useful, and in my experience any sub-4 quants get ridiculously incoherent, which is why those models tend to try to 'wow' people with tok/s rather than benchmarks.

haven't tried it yet though.

by cyanydeez12 hours ago|

parent|

[-]

Qwen3.8-Flash-Next fits in 80GB on strix halo. That's when I started real "vibe" coding with them. Speed is still an issue if I want to move fast and break things, but I've built a little universe for development (I assume many people have also) and it's cruising.

But yeah, 32-64 is the gateway I think. If Qwen4 can make a leap in the flast-next MoE and hit 64 GB on VRAM w/context, that's definitely the cloud killer to me.

by Tepix14 hours ago|

parent|

prev|

[-]

I’m wondering myself: Do bigger models suffer less from quantization? Are there papers?

by ricardobeat13 hours ago|

parent|

[-]

They seem to degrade a lot less.

You don't need papers, good quants are usually published with benchmark data, for example: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-... which maintains parity with FP8 even at 2 bit.

by alfiedotwtf12 hours ago|

parent|

[-]

Always check the label and if pain persists, see your doctor!

The ISTA accuracy looks good on that page, but it's only a chosen handful. Instead, look at most of the graphs on Unsloth's Qwen 3.8 Flash Next - https://unsloth.ai/docs/models/qwen3.8-next

As always, check the quant vs KLD graphs (half way down that page).. I found that not all quants are the same - Unsloth's GLM Q4 -> Q3 degrades a lot faster than their Minimax M3.1 Q4 -> Q3 ggufs.

by pitchlatte13 hours ago|

parent|

prev|

[-]

you can get a lot done with models well under that - the google and qwen 20-30B class models are fantastic

by rpdillon11 hours ago|

prev|

[-]

Deepseek v4 flash 0731 in a 2-bit quant has been useful for me. Other models don't seem to hold up nearly as well at 2-bit. Most of my other models are 5-6 bit quants because of this.

by LeBit13 hours ago|

prev|

[-]

This but also the kv quantization shouldn’t be lower than 8 bits.

by mt_13 hours ago|

prev|

[-]

Next frontier is achieving good output with that kind of quant dimension

by ranger_danger12 hours ago|

prev|

[-]

There are solutions for that: https://prismml.com/news/bonsai-2-27b

(and before you crap on it, this is not the same as the original Bonsai, this newer one is much better, and upstream support in llama.cpp etc. is being worked on)