upvote
The most important feature of a tokenizer is dividing the inputs into independent values that the neural network can work with. It's not the size.
reply
I mean it might lead to better performance on the model side. So the tokenizer is better but more expensive.
reply
It’s better for them
reply