undefined

points

by Tactical4515 hours ago |

comments

by XCSme15 hours ago|

[-]

I don't think it matters, if it's for local/on-device usage.

The cost is similar vram footprint I guess (?)

by sleepyeldrazi13 hours ago|

parent|

[-]

loading the model would be similar vram footprint, correct, however the size of KV is based on 'active' params, not total params. So while at 1k ctx both will be in the same ballpark vram footprint-wise, at 100k the story will be very different. 27B at q4 kv for 256k takes ~8gb, while 35B at q4 kv around ~3.5gb, so at full precision kv those would be ~32gb and ~14gb (all ballparks, if you want exact numbers, its not hard to test).

As for the "cost", here i think the interesting arguments are around speed vs accuracy/"getting the job done", not literal $ cost per token.

by XCSme13 hours ago|

parent|

[-]

I am asking mostly for running on a 3090.

I think the tps difference between them (both fitting in vram) won't be more than 2x in practice.

I would happily take 20tps over 40tps, if the model gets 3x more correct answers.

by sleepyeldrazi13 hours ago|

parent|

[-]

Speed is (for the most part) active-parameter based, so a 30B-A3B model is roughly 10x the speed of a dense 30B (realistically closer to 8x) in the case when both fit. That's the proposition of MoE and why everyone is trying to make massive models with very few active params, so that they are still fast while having access to a lot of knowledge (at the cost of reasoning, as reasoning ability 'for the most part' comes from active params).

You can test this by running this nemo or 35B on the 3090. I have and its very fun (but sadly a worse model than 27B, so I usually keep 27B on my 3090)

by XCSme12 hours ago|

parent|

[-]

I remember running both qwen 30b-a3b and 27b on my 3090, and on the initial test, the 27b was only like 2x slower.

by sleepyeldrazi3 hours ago|

parent|

[-]

Ran a quick test so that we both have accurate numbers, without MTP* at 10k ctx 27B hovers around 42 ts in llama.cpp, 35B around 135 ts. So not the 8x I assumed, just over 3x, but thats still a big difference.

For the sake of testing I turned MTP off, as that heavily depends on what the generated text is (structured text like code is very often a lot more predictable, therefore bigger boosts) and the quality of the quantization, as drafters learn how to "mimic" the full precision generated tokens, so when you layer the fact that MTP is a 'guesser' of the main model's next token, and quantization affecting what exact token is generated, it'd make comparisons like this needlessly noisy.

by XCSme2 hours ago|

parent|

[-]

Thanks for sharing.

Yeah, that was my experience too (2x or 3x is indeed considerably faster), but not workflow-changing faster at 45tps baseline, especially for asynchronous tasks (which is my goal with a local 3090, to just let it do things non-stop, without my intervention).

What was the result with MTP?

Isn't MTP "losless"? The result should still be relevant when averaged across a fee queriers across different domaine I guess.

by sleepyeldrazi36 minutes ago|

parent|

[-]

MTP is lossless in the sense that running a model with and without (at temp=0, meaning no randomness) will produce identical results. It's true that with enough samples across domains and runs with MTP it should even out around concrete numbers, but I don't have time currently for long tests. On a quick test (before I remembered MTP is on), 27B was around 60-70 ts and 35B around 180-200 ts, both going up and down but mostly in those ballparks, which is inline with the ~3x from not using MTP.

One somewhat related thing is that, without drafters (the models doing just generation) ts tends to slowly go down, while with drafters, as the model is "checking drafts" instead of pure generation, even when the avg ts has fallen to say 50 ts (27B, ~128k ctc for example), there still tend to be random spikes to up to 70 (mostly during coding), which is always funny to me but also makes models a bit more "resilient" to the slow degradation of ts.