Hacker News
new
past
comments
ask
show
jobs
points
by
not-a-llm
6 hours ago
|
comments
by
nullsanity
6 hours ago
|
[-]
So, worse? Because we benchmark off token use when talking about token use, and everyone else understood that.
reply
by
marcosdumay
2 hours ago
|
parent
|
next
[-]
The most important feature of a tokenizer is dividing the inputs into independent values that the neural network can work with. It's not the size.
reply
by
jascha_eng
4 hours ago
|
parent
|
prev
|
next
[-]
I mean it might lead to better performance on the model side. So the tokenizer is better but more expensive.
reply
by
otikik
5 hours ago
|
parent
|
prev
|
[-]
It’s better for them
reply