I see people give too much importance to specific engineering design choices of the current generation of LLMs. Tokenizer is not an absolutely essential part of the system. It’s just and adapter for text input/output. It can be eliminated completely and model can use bytes directly.
replyI think the short story captures this well. Weights (connections) are the essential and philosophically important part. They do the thinking, memory, singing etc.
A tokenizer is roughly and approximately Huffman-coding sequences of input (bytes of English etc) into shorter sequences (list of tokens), as a performance optimization.
replyAs you said, it's not in any way intrinsic to the LLM, though it may be a very necessary optimization on today's hardware.
I wouldn't use the word necessary.
replyIMO, we are probably talking about a 6x slow down (for typical english). You would need to be absolutely stupid not to implement some kind of optimisation along these lines.
Slower and maybe a little dumber; But it would work.
Not sure about “dumber” - it may be better than SOTA models at identifying which days of the week contain the letter “d”.
reply