PS: only now noticed that 76.87GiB is 82.5GB, which happens to be size of the unsloth IQ1_S. The “technical” paper also mentions IQ1_S at some point. Sounds to me like the author’s model or the author mixed IQ2_XXS with IQ1_S at some point.

Also, just noticed that “moe pruning” likely means “the model is supposed to run 6(DS)/8(Qwen) experts per token per layer, but we only run 4 to save on time”.

Either way, the final gguf is 75% of the size of unsloth IQ1_S, which by itself should tell you a lot about its quality; they (somehow; I have no interest digging that info from that slop paper) shrink the (already IQ1) FFN weights, and shrink some attention weights from Q5 to Q2. I’m surprised it was even able to generate the text in the demo.