upvote
Yeah fair enough, maybe religion vs science wasn't the best example to use. It was the example I could come up with off hand.

> The objective is to predict, at minimum, internet scale text.

But then what does it compress exactly? How do you compress the answer to a question that has multiple answers? How do you pick "the right one"? That's at least what I get from this idea of compression that there is some congealing of information.

reply
An adjacent comment explains your misconception about the nature of what's being described as compression.

Separately I'd like to add that you seem to be focused on the concept of some sort of universal truth. Meanwhile compression has nothing to do with that. Input is entirely arbitrary - it can contain contradictory statements or even random noise. Obviously truly random noise isn't compressible but for example pseudorandom values derived from a seed are. At its core compression is merely finding a set of rules that can be used to describe the data in question.

reply
> But then what does it compress exactly?

Tokens into dimensions.

A good example is how humans talk about "left and right" in terms of political systems.

Humans are able to intuitively place these complicated multivariable economic-political systems into a one dimensional line that we call left or right wing.

Of course, this deep compression complicates things... why is national socialism considered right, while democratic socialism is considered left.

You can get more signal by increasing dimensions, so let's say instead just left or right, we have 2 axis one for economic and one for political. This would make things more clear. Even clearer if we add another dimension, let's say authoritarian or democratic. We can add yet another, socially liberal or conservative.

You can keep going, and as you add more dimensions it makes things clearer / more intelligent (to a limit.)

AI is doing this to text. It ingests text, and creates a token vocabulary. IIRC most models have a vocab of hundreds to low thousands of tokens.

This token vocabulary (how many distinct tokens it uses) is how many dimensions the AI thinks in, so to speak.

After this, there is the whole training process of how tokens relate to each other.

So your question of

> How do you compress the answer to a question that has multiple answers?

Isn't applicable, as it doesn't compress anything we would call a question or answer.

Instead, answers to questions are an emergent property of how text fragments relate to each other, and is why it can give contradictory answers.

reply
You don’t - what’s reported is average compression, not for the particular examples you made. Some things are more compressible than others
reply