Ive always thought that Serbian is the worst off out of all European languages. Both cyrillic and latin script are widely used. Latin also has a naked variant, where all diacritics are left out (which is especially prevalent online). So all Serbian text is in a three-way split between these scripts.

Also, it is non-EU, so initiatives like [0] will only indirectly help with representation (Croatian being the same* language, but it does not use cyrillic script)

* Contentious point maybe, but they are mutually intelligible with what would otherwise be considered only dialectical variations

[0] https://digital-strategy.ec.europa.eu/en/news/commission-wel...

Aren't tokenizers a hack to turn Latin alphabet texts into a pseudo Chinese with better per-unit information density?

logographic languages still win I guess, a 4-byte chinese character beats a n-character English word for that same concept.

but yeah, anything alphabetic yet not latin gets the worst tradeoff.

> It's just that byte pair encoding naturally favours English and other Latin languages because they are dominant in the training data.

Not really, because I don't think anyone use the whole training dataset to train the tokenizer.

From my brief (disclosure: claude-assisted [0]) internet search I learned that typically they use from tens to hundreds GB of data to train the tokenizer. Sometimes it is hand-curated, sometimes it is randomly sampled.

Of course, you can deliberately choose how much text in each language do you include in the training set for the BPE and deliberately optimize for (or against) a specific language.

[0] https://claude.ai/share/da277189-f23a-4071-bd9f-2d75fe6aa779

They are typically optimized for English, for example, they often use rules for English contractions. See this thread: https://x.com/_albertgu/status/2113687798826111283

I don't know whether European languages work inherently better with tokenizers, but I don't see why they would.

I've done my own research on that as a part of my Aqueous [1] archiver, and the conclusion is that the inherent information entropy is about as same (CJK scripts, for example, compresses much better when they are turned into their native character encodings). So a script-aware tokenization should make that irrelevant.

[1] https://lifthrasiir.github.io/aqueous/ (specifically wrt-recons-hangul codec)

Are bytes relevant? These things are billed by the megatoken, not the megabyte.

utf8 is just one of the encodings that exist, and is kind of irrelevant at the point it contacts the llm

bit configurations aren't uniform across those bytes, everything should compress to similar sizes