upvote
I don't know if it's fair to say that a tokenizer is being less efficient if it generates more tokens per text. I think it's more fair to say the tokenizer is more nuanced. The question is whether the additional nuance permits better model output, which could justify the additional token cost in inference.
reply