upvote

points

by lostmsu1 days ago |
comments
upvoteby jakswa19 hours ago|
next
[-]
I'll back up your smaller claim, but be specific that it's UD-Q4_K_XL size:

- muse glimmer: 15.9GB

- qwen 3.6 27B: 17.6GB

My video card is so close to its limit that these GB thresholds are mattering too much for me :D

reply
upvoteby IsTom1 days ago|
prev|
[-]
How is 30B smaller than 27B?
reply
upvoteby LeBit1 days ago|
parent|
next
[-]
It uses fractal compression
reply
upvoteby lostmsu23 hours ago|
parent|
prev|
[-]
They say it is trained with quantization awareness, so it should only be 15GB or so. Qwen was only trained in FP8 with QAT.

UPD, NVM, got misled by comments here. It is actually almost 60 GB so much larger

reply
upvoteby ricardobeat22 hours ago|
parent|
next
[-]
Quantization awareness doesn’t change the size of the weights, just means it won’t degrade when quantized. QAT = quantization aware training. They will both be very similar in size at the same quant.
reply
upvoteby xienze23 hours ago|
parent|
prev|
[-]
You're mixing up sizes of different quants. The 60GB is unquantized, and Qwen's unquantized size is around 54GB. Their sizes as like quantization levels are similar.
reply
upvoteby lostmsu21 hours ago|
parent|
[-]
From my perspective it doesn't make sense to talk about the number of parameters. What matters is model size in bytes and its performance at that certain size.

Meta actually relesed official 4 bit quants in 17GB, but I haven't seen any indication that training was quant-aware, so the quants are not going to have same performance. 3.6 27B has official FP8 quant that AFAIR was trained with quantization awareness.

The best example is last year's gpt-oss which was released prequantized in mxfp4 so 20B parameter model was under 14GB and 120B was under 70GB right away.

reply
upvoteby petu20 hours ago|
parent|
[-]
> but I haven't seen any indication that training was quant-aware

readme on huggingface says they've benchmarked the quants -- for 17GB quant reported 1% avg loss across 15 benchmarks (sadly no breakdown).

I assume that's strong enough signal for QAT. Not just first party quants, but they cared to monitor degradation.

reply
upvoteby lostmsu19 hours ago|
parent|
[-]
> sadly no breakdown

That's exactly the point. We know short context knowledge stuff does not regress with quantization. But I expect agentic intelligence to suffer greatly.

If I were to pick one bench, I would like to compare quants on TerminalBench Hard. But then Glimmer already loses to 3.6 27B on it by a large margin.

reply