Ive always thought that Serbian is the worst off out of all European languages. Both cyrillic and latin script are widely used. Latin also has a naked variant, where all diacritics are left out (which is especially prevalent online). So all Serbian text is in a three-way split between these scripts.

Also, it is non-EU, so initiatives like [0] will only indirectly help with representation (Croatian being the same* language, but it does not use cyrillic script)

* Contentious point maybe, but they are mutually intelligible with what would otherwise be considered only dialectical variations

[0] https://digital-strategy.ec.europa.eu/en/news/commission-wel...

Aren't tokenizers a hack to turn Latin alphabet texts into a pseudo Chinese with better per-unit information density?

logographic languages still win I guess, a 4-byte chinese character beats a n-character English word for that same concept.

but yeah, anything alphabetic yet not latin gets the worst tradeoff.

> It's just that byte pair encoding naturally favours English and other Latin languages because they are dominant in the training data.

Not really, because I don't think anyone use the whole training dataset to train the tokenizer.

From my brief (disclosure: claude-assisted [0]) internet search I learned that typically they use from tens to hundreds GB of data to train the tokenizer. Sometimes it is hand-curated, sometimes it is randomly sampled.

Of course, you can deliberately choose how much text in each language do you include in the training set for the BPE and deliberately optimize for (or against) a specific language.

[0] https://claude.ai/share/da277189-f23a-4071-bd9f-2d75fe6aa779

They are typically optimized for English, for example, they often use rules for English contractions. See this thread: https://x.com/_albertgu/status/2113687798826111283

I don't know whether European languages work inherently better with tokenizers, but I don't see why they would.