The problem with the rest of inference is that changes are not trivially correct or incorrect, as they are with the tokenization layer.
replyEh, linear algebra changes are still easy to measure correctness, it's just that you're competing with 50 years of research for most of them, less low hanging fruit.
replyI'm sure there's been a lot more effort put into the other, more consequential, portions of inference time.
replythe answer is many! This would take hours to write. Full teams and research on nearly every part. So many 'unlocks' coming.
reply