> For decode, (...) it is difficult to batch multiple output tokens together.
replyI think it's the other way around? The GPU has to stream gigabytes of active layer weights to compute the next token, so having a batch of next-token predictions sitting there on the GPU goingh through the layers makes better use of the bandwidth.
At least that's what I observed on a Strix Halo, batching 4 predictions yields like 2-3x the total tps.
MTP can give you small batches, but it is still WAY smaller than the batches you can get with prefill, which is limited only by the number of input tokens you have (but has diminishing returns on performance).
replyBut:
1) It still makes no sense to upload the weights to the GPU with MTP as you are still bottlenecked by the weight upload.
2) I'm not sure MTP helps much with MoE models.
deleted
reply