Indeed. If you're going for a catchy generalization, at least write it correctly. Most compression is history, and only extrapolates under the assumption that "nothing changes".
replythere are dictionary compressors (decent compression, most common, fast), and statistical compressors (better compression, slower). Statistical compressors are much closer to LLMs in that an llm is learning statistics about the data too.
replyAnd yes, compression is history, that's what statistics are all about. Statistics can only measure the past to make a prediction about the future. And LLMs work in the same way. The context is the history, and given that history, it predicts the next token. An LLM can, almost trivially, be dropped into something like the PPM statistical compressor (it's just replacing one implementation of a markov chain with another).
Anything can only represent past measurements. Statistics is not an exception. But they don't make a prediction about the future. That comes from a model you have, and it often is implicit: "the linear trend from the last 12 months will hold in the next month" or whatever. So compression isn't by definition prediction. The other way around doesn't have to hold either, but in the case of LLMs it does.
replydeleted
reply