upvote
You can compress syntax, losslessly even, with zero understanding of its semantics. Zero understanding not only imbued into the compressor/decompressor, but even the designer of the compressor doesn't require understanding the semantics. Actually, even of the syntax.

A compression program can compress a book written in a language that the author of the program doesn't understand, on a topic he knows little about.

reply
Finding common characters and building a list of words is a low level type of understanding. Doing it better does actually start directly representing syntax patterns and that's a less-low level of understanding.

I think "losslessly even" is the wrong way to think about it. Lossless compression often requires less understanding than high quality lossy compression. If you can do a lossy compression that correctly decides what details are unimportant, that's a good sign of understanding.

reply
> I think "losslessly even" is the wrong way to think about it. Lossless compression often requires less understanding than high quality lossy compression. If you can do a lossy compression that correctly decides what details are unimportant, that's a good sign of understanding.

This is the crux and reminds me of things like mp3 that exploit the nature of human hearing being limited to a frequency range.

reply
We model the data. The model, hopefully, captures something real in the data. If it does, then it's fair to say that we understand the data better.

But it's frankly a philosophical question what's real or not. No model is going to capture absolutely everything about the thing it models - at that point, it would be the thing. The best we can hope for is that it captures everything we care about.

And no experiment or metric can tell you if you care about the right things. At best it can tell us if we care about a thing given other things we care about. "No cares in, no cares out".

To make it a little more concrete: you could compress a string from back to front. You could build an LLM to help you do that. If you care about file size, that's almost certainly a bad idea, the forward LLM will be better for that purpose. But are there purposes for which the backward LLM might be better? I think that's not so hard to imagine. Often we wonder about "what came before".

reply
A model will not capture every nuance of what it models, but it might capture every nuance you managed to measure properly.
reply
Including the things you didn't even realize you measured, or even conceptualized at all.
reply