Hacker News
new
past
comments
ask
show
jobs
points
by
ben_w
5 hours ago
|
comments
by
paulryanrogers
5 hours ago
|
[-]
If tokens use UTF-8 bytes then it could be a factor.
reply
by
ben_w
4 hours ago
|
parent
|
[-]
A token is an index into a dictionary. What the dictionary uses isn't seen, this is why they couldn't count the 'r's in 'strawberry'.
reply