Hacker News
new
past
comments
ask
show
jobs
points
by
logicallee
10 hours ago
|
comments
by
frwickst
10 hours ago
|
[-]
Since the output is quite long, here is a link:
https://pastebin.com/k76wiVGP
reply
by
hrimfaxi
10 hours ago
|
parent
|
[-]
Why does this G character appear to prefix most of the output? ("Ġlike")
reply
by
frwickst
10 hours ago
|
parent
|
next
[-]
It is a tokenizer artifact most likely (
https://github.com/huggingface/transformers/issues/4786
). So the output is not properly decoded in this case, it should just be a space.
reply
by
kgeist
10 hours ago
|
parent
|
prev
|
[-]
The original tokens have Ġ instead of space. I had this issue too when writing an inference engine for Qwen. You have to "normalize" those special characters.
reply