For the sake of testing I turned MTP off, as that heavily depends on what the generated text is (structured text like code is very often a lot more predictable, therefore bigger boosts) and the quality of the quantization, as drafters learn how to "mimic" the full precision generated tokens, so when you layer the fact that MTP is a 'guesser' of the main model's next token, and quantization affecting what exact token is generated, it'd make comparisons like this needlessly noisy.
Yeah, that was my experience too (2x or 3x is indeed considerably faster), but not workflow-changing faster at 45tps baseline, especially for asynchronous tasks (which is my goal with a local 3090, to just let it do things non-stop, without my intervention).
What was the result with MTP?
Isn't MTP "losless"? The result should still be relevant when averaged across a fee queriers across different domaine I guess.
One somewhat related thing is that, without drafters (the models doing just generation) ts tends to slowly go down, while with drafters, as the model is "checking drafts" instead of pure generation, even when the avg ts has fallen to say 50 ts (27B, ~128k ctc for example), there still tend to be random spikes to up to 70 (mostly during coding), which is always funny to me but also makes models a bit more "resilient" to the slow degradation of ts.