Exploring compression algorithms for weights is a good idea, and I hope you have a successful product. However, if you can prove this statement:
reply> reduces it down to its minimum entropy -- it cannot be compressed further.
I think you could make a lot more money elsewhere :-)
https://en.wikipedia.org/wiki/Kolmogorov_complexity#Formal_p...
We're not an AI company ... nor do we have any reason to use it. Just a fun idea that was fruitful.
replyAnd to clarify -- it only applies to models, not arbitrary data. So its useful to exactly zero other fields.
replyThat's very interesting. Does that mean you can reduce say, a 30B class Q8 from ~30 GB down to 10 GB or less?
reply704gb -> 564gb; 358 gb -> 270 gb; 28.79 gb -> 7.65 gb; 439 gb -> 93 gb
replyIt depends on the total entropy of the model. Smaller models have less entropy.
> Smaller models have less entropy.
replyInteresting. Why is that? I would have expected the opposite, since larger models have to try less hard to fit the training data. Or maybe this leaves more parameters with random initialization, resulting in higher entropy for larger models?
I honestly don't know... I didn't train the models, so I can't tell why the math works out that way. It just does. I suspect it has to do with the fact that all the small models I've tested have been quantized. I don't know of any small model trained from scratch. If you know of any, I'd be happy to encode it and see what it looks like.
replyWe have a lossless compression codec (working on open sourcing it over the next couple of weeks) that reduces it down to its minimum entropy
replyLOL
deleted
reply