I feel like even TPS is becoming less of a good metric as we're seeing certain models handle similar problems while burning far fewer tokens.
reply> I don’t really care about the CPU GHz and the memory bandwidth ... What I really want to know is how many tokens per second this will deliver
replyThere is a fairly direct link between the two numbers. You can predict the latter from former reasonably well
I always heard of GB/s as most important number ... AI told me their 8800 MT/s on 8 Byte, 12 DDR5-Channels means 845 GB/s. A Nvidia RTX 4090 has 1008 GB/s. Nvidia B200 has 8000 GB/s.
Is this the right way of looking at it?
replyYou have to load the model weights into VRAM over PCI-E (from RAM). So the (PCI-E) bandwidth strongly affects time to first token.
replyYou have to run inference on the GPU by reading and writing to VRAM. So TFLOPS of the compute matters, and bandwidth to the VRAM (Always integrated with the GPU, rarely a bottleneck), and this strongly affects tokens/s
If you're doing training workloads or offloading to system RAM, it gets more complicated. (And mostly bound up trying to feed compute on time)
(Edits for clarity.)
> So the (PCI-E) bandwidth strongly affects time to first token
replyOn dedicated inference hardware I'd expect model weights to never leave the RAM, and you'd probably load them on startup before even starting to serve requests
[flagged]
reply