Afaik this is already a part of regular quants, Q4 just means the average reduction in accuracy is at that level for the whole model, some weights will be fp16, some int8, some int4, some even lower. 4KM is slightly larger than 4_0 for example.
replyReally? I didn't know, where does it spec this usually?
replyI have never seen it neither on the link above nor on HF.
On HF, if you look at the model metadata, practically all of the rank one tensors (or stuff like the input embedding tensor where it's only multiplied by a one hot, which becomes a O(1) select rank 1 tensor from array practically) normally gets stored as fp32.
replyOllama will expose the metadata on models too, showing the quantization for each individual tensor.
Using tooling built into llama.cpp you can quantize a model yourself into a flat q4 bpw
replyOther techniques likely beyond the scope of that one utility allow you to quantize some weights more and some less. So most model teams put our "average bpw" numbers.
What do you mean never seen?
replyOn HF, click on any Q2K quantized model in GGUF format, it will show the metadata. Look at the expansion of the blocks. You will a mix of quantizations.