Hacker News
new
past
comments
ask
show
jobs
points
by
gavinsyancey
7 hours ago
|
comments
by
dosinga
37 minutes ago
|
[-]
roughly the tokenizer is optimized such that the training data (for the tokenizer) can be represented with the smallest number of tokens. and yes, English tends to be overrepresented in training data.
reply