I've done my own research on that as a part of my Aqueous [1] archiver, and the conclusion is that the inherent information entropy is about as same (CJK scripts, for example, compresses much better when they are turned into their native character encodings). So a script-aware tokenization should make that irrelevant.
reply[1] https://lifthrasiir.github.io/aqueous/ (specifically wrt-recons-hangul codec)
Are bytes relevant? These things are billed by the megatoken, not the megabyte.
replyIf tokens use UTF-8 bytes then it could be a factor.
replyA token is an index into a dictionary. What the dictionary uses isn't seen, this is why they couldn't count the 'r's in 'strawberry'.
replyutf8 is just one of the encodings that exist, and is kind of irrelevant at the point it contacts the llm
replybit configurations aren't uniform across those bytes, everything should compress to similar sizes
reply