Hacker News
new
past
comments
ask
show
jobs
points
by
dannyw
22 hours ago
|
comments
by
16 hours ago
|
next
[-]
deleted
reply
by
stymaar
15 hours ago
|
prev
|
next
[-]
Looks like we have a /r/localllama dweller here.
reply
by
venusenvy47
21 hours ago
|
prev
|
[-]
Where do you find this information for each model?
reply
by
adrian_b
18 hours ago
|
parent
|
next
[-]
When you look on HuggingFace.co at the files of a model, for each model you will see a file "tokenizer.json".
In that file you can see all tokens and their corresponding numeric codes.
reply
by
ComputerGuru
21 hours ago
|
parent
|
prev
|
[-]
The tokenizers are included in the open s̶o̶u̶r̶c̶e̶ weights releases; you wouldn’t be able to use the weights without the corresponding encoder/decoder, in fact.
reply