The model is natively quantized (i.e. it was trained that way in the first place, so this is not a post-training quantization which degrades performance).
replyIsn't it not completely quantized? I thought there were some dense parts but most is int4?
replyBut the huggingface link mentions BF16, F16, and I32?
replyNot every weight is quantized. For example, those weights which don't take much space or are highly important are left in higher precision. State-of-art quantization of weights is never done uniformly (i.e. to all weights and in the same way).
replyI don't believe safetensors has a native int4 dtype, so they packed 4 int4s into a bf16 in this checkpoint.
reply