Hacker News
new
past
comments
ask
show
jobs
points
by
computably
8 days ago
|
comments
by
fooofw
8 days ago
|
[-]
The tokenization can represent uncommon words with multiple tokens. Inputting your example on
https://platform.openai.com/tokenizer
(GPT-4o) gives me (tokens separated by "|"):
lower|case|un|se|parated|name
reply