Hacker News
new
past
comments
ask
show
jobs
points
by
kfsone
3 hours ago
|
comments
by
mips_avatar
1 hours ago
|
next
[-]
Using ascii characters is a simple form of tokenization with less compression
reply
by
b44
3 hours ago
|
prev
|
[-]
hm. the way i see things, characters are the natural/obvious building blocks and tokenization is just an improvement on that. i do mention chatgpt et al. use tokens in the last q&a dropdown, though
reply