undefined

points

[-]

I think your leading statement is a bit more authoritative than what the linked paper actually demonstrates. It demonstrates your claimed qualities for specific algorithms, methodologies, and models.

It explicitly says that Quantization-Aware Training is out of scope, may reveal different trends and has potential for extreme quantization. Heavily-quantized model releases today generally use QAT or something similar, so I don't think the paper can be universally applied to modern quants (e.g. [0]).

That said, I don't see any evidence that TFA does any of that so the paper's claims may be quite applicable there.

[0]: https://huggingface.co/vcruz305/DSV4.1-Flash-1017-SAGE-REXL3...

by bitexploder10 hours ago|

parent|

[-]

And things like GSQ-RCO optimization from ISTA labs, EXL that you linked. Preserving bits for certain (large) domains like agentic coding. Pruning others. It really works and creates functional models.

by fho11 hours ago|

parent|

prev|

[-]

I recently did some toying around with (very) small language models (<200k). I used QAT from the beginning and was surprised that I got byte identical results from the int8 model vs the float32 one.

If that scales with parameters I don't see why QAT shouldn't be the norm.

(As I was only comparing the QAT f32 model vs the quantization its possible that a not QAT f32 would have performed even better)

by lostmsu10 hours ago|

parent|

[-]

Do you think QAT isn't the norm yet?

I expected it to be the norm since GPT-OSS

by onlyrealcuzzo13 hours ago|

prev|

[-]

I imagine that it's just the non-linearity of audio makes it easily apparent how much quantization really matters, and that benchmaxxing is so ingrained in LLMs that if you simply run them through benchmarks, they can do well under quantization, when in reality, they have a similar drop off that they do in audio, it just doesn't slam you in the face at every turn as obviously.

by pu_pe9 hours ago|

prev|

[-]

There is always quality degradation when quantizing, but some models keep surprisingly good performance at aggressive levels of quantization too. DeepSeek v4.1 flash at 2-bit is still a beast for example. In my own experience I would easily trade off accuracy/intelligence for a vast speedup, especially if your setup simply wouldn't run something at 4-bits.

by zobzu8 hours ago|

parent|

[-]

sometimes yes sometimes no :)

that's why different models exist.

by augment_me9 hours ago|

prev|

[-]

This is not fully true. 4 over 6 paper(https://arxiv.org/pdf/2512.02010) for example showed that the initial and last layers of transformers that expand into the high-dimensional vocabulary are hurt much more than the center ones. These layers are kept in fp16 and are hurt much more than others by quantization.

If you think of it from an informational perspective, you can imagine that text tokens are much more high-dimensional than a transformed/down-projected variant.

With this reasoning the central layers are mostly building onto the residual stream, and can do fine with 1b and 2b weights.

by moffkalast8 hours ago|

parent|

[-]

Afaik this is already a part of regular quants, Q4 just means the average reduction in accuracy is at that level for the whole model, some weights will be fp16, some int8, some int4, some even lower. 4KM is slightly larger than 4_0 for example.

by augment_me8 hours ago|

parent|

[-]

Really? I didn't know, where does it spec this usually?

I have never seen it neither on the link above nor on HF.

by monocasa7 hours ago|

parent|

[-]

On HF, if you look at the model metadata, practically all of the rank one tensors (or stuff like the input embedding tensor where it's only multiplied by a one hot, which becomes a O(1) select rank 1 tensor from array practically) normally gets stored as fp32.

Ollama will expose the metadata on models too, showing the quantization for each individual tensor.

by julianlam8 hours ago|

parent|

prev|

[-]

Using tooling built into llama.cpp you can quantize a model yourself into a flat q4 bpw

Other techniques likely beyond the scope of that one utility allow you to quantize some weights more and some less. So most model teams put our "average bpw" numbers.

by kanbankaren8 hours ago|

parent|

prev|

[-]

What do you mean never seen?

On HF, click on any Q2K quantized model in GGUF format, it will show the metadata. Look at the expansion of the blocks. You will a mix of quantizations.

by liuliu4 hours ago|

prev|

[-]

It is a native 4-bit model, so you would think a 2-bit roughly equivalent to 4-bit of a native bf16 model.