Most quantized GGUFs have a very high Shannon entropy, so are almost perfectly compressed already. A Q4_K_M quantization is often around +7.90 bits/byte and so is virtually incompressible.
Unquantized models (F16) are often around +7bits/byte for comparison.
https://en.wikipedia.org/wiki/Shannon%27s_source_coding_theo...
I'm not too familiar with LLM memory access patterns, but since the model files are stored on disk, you wouldn't ever keep them in memory anyway (except caching) and just reread them on demand.
Dense architectures hit every byte of the model for every token. It's the ultimate memory bandwidth nightmare.
MoE models use subsections for each token and are more amenable to keeping parts in secondary storage, and indeed there are some local llm engines that do this.
One way to think about it is that if the weights were compressible, it’s a clear signal that better training could make the model smarter as there’s redundant information in the structure that could be used to store other meaning.
The other challenge with compressing weights of course is that you have to decompress them before you do math with them. Dequantizing weights is significantly cheaper and supports random access too (not that that matters too much but it helps in terms of leveraging SIMD techniques to do it in an embarrassingly parallel way)
Consider the induction heads, their W_O are generally expected to be the identity matrix and it is useful to have many of them at many layers (e.g. if you only had one induction head in the whole network you'd only be able to perform one induction circuit per token emitted)
That being said the attention heads are a small portion of the total network size
Interesting comparison, I wonder if there are loss-less compression schemes that are "weight-aware", about the statistical properties of model weights.
Back to that line - because (Huffman coded) JPEG images can be lossless recoded using arithmetic coding, which makes them smaller than if they were compressed with the best general-purpose compression algorithms/tools (e.g. `jpegtran -arithmetic < input.jpg > output.jpg` beats `xz -9 < input.jpg > output.jpg`). AC takes advantage of the properties of the image coding better than general-purpose compression tools on the output of the Huffman coding.
But all seem to have massive issue boards. What I did see was oMlx release clustering so you can string 3 macs and have 3 times the memory or use one for prefilling accelerating.
Which I thought was neat and what I'll likely poke at instead of getting more ran.
So my mind on that.