undefined

points

by mdp20219 hours ago|

[-]

> Better... than 3.8 27B

How can a "6b active per token" MoE be better than a just released 27b in the same family? Possible maybe, but quite interesting and requiring some explanation.

Edit: ok, on second thought, probably because the small "experts" are really rich in specialized training compared to the dense model. Still raising questions about the details, e.g. the reasoning abilities (or all meta-skills) of a "6b per token" network compared to the dense capabilities...

by hedgehog8 hours ago|

parent|

[-]

Separating knowledge from reasoning so you only pay for what you use is a big rationale for MoE, the big problem being MoE training has historically been hard to get right. In a dense model every single token you're paying a cost to determine whether you're now talking about the flavor of durian.

by mdp20216 hours ago|

parent|

[-]

Sure, the "6b subset" can be more knowledgeable on its area than a whole 27b generalist (and more efficient), but where is the simulated Intelligence encoded? A 6b subset as or more intelligent than a 27b raises the question of how metacognition skills are stored.

by mapontosevenths4 hours ago|

parent|

[-]

MOEs are built by training a second "router" model to identify which parts matter inside the dense model.

Think of MOE as ignoring noise, rather than a more efficient encoding of the data we teach it which ones can be ignored. Turning down the noise actually sharpens the results sometimes.

Modern LLM's are wildly inefficient.

by data-ottawa9 hours ago|

prev|

[-]

We should still expect significant performance improvements.

I believe Unsloth’s branch (and ggml’s) don’t support MTP or the ngram embedding yet.

by hedgehog8 hours ago|

parent|

[-]

The active param count is so small I'm not sure how much advantage MTP will have, the current llama.cpp does load the ngram embeddings but I haven't verified it uses them. I expect to redeploy all this stuff every few days as the tooling gets improved.

by cmrdporcupine8 hours ago|

parent|

[-]

My initial implementation of MTP I have here in my own (DGX Spark specific) custom runtime brought it up from ~12 tok/sec without MTP to ~16 to ~20 with; depending on workload.

It's not world changing, but at those speeds I'll take anything I can get.

(The ngram embeddings in this case are paged to/from disk which seems to cost ... basically nothing).

https://github.com/rdaum/eider/

by hedgehog7 hours ago|

parent|

[-]

It should be faster, Q6 on Ryzen 395 using Vulkan llama.cpp is about 22 tokens/s with no MTP, and I'd expect the Spark to be 20% faster or something in that neighborhood.

by cmrdporcupine7 hours ago|

parent|

[-]

Probably. I've spent zero time with optimization at this point. Code is all new this morning.

Curious if llama.cpp is doing full BF16 for the n-gram embeddings table, or if that's quantized, too. I was going to try nvfp4 for that but wasn't sure about the quality risk.

Spark usually wins on prefill, not decode. I think memory bandwidth is about same between the two. What do you get for prefill/prompt?

by hedgehog7 hours ago|

parent|

[-]

Ok, at 50k context its about 126 prefill, 13 generation.

by cmrdporcupine7 hours ago|

parent|

[-]

Oh, interesting. So beats on prefill and matches at decode. MTP will help you a bit once you have it. I've got a lot of work to do to optimize prefill.

Kinda wish I had a Strix Halo here to play with as well.

by hedgehog4 hours ago|

parent|

[-]

I just took a minute to look at your eider repo, very cool. It looks like most of the code outside the kernels and immediately surrounding plumbing would work well on AMD APUs, and probably also on Apple and newer Intel.

by cmrdporcupine3 hours ago|

parent|

[-]

Thanks; I have another, currently private, repo that targets both pure CPU inference and wgpu (to vulkan.) It's somewhat similar but... different. Shares some common pieces but needs to be refactored to share more.

But it's been hard to make it competitive with CUDA. At least on this Spark and my only non-NVIDIA machine (which only has 16GB unified relatively slow RAM.)

by hedgehog2 hours ago|

parent|

[-]

I know a little bit about this problem space from previous work (we were working on performance-portable deep learning back around 2016). The infrastructure has improved but as far as I can tell not many teams have really "squeezed the toothpaste tube" and worked through performance issues systematically. These days a small team and robots can probably do it though.

by cmrdporcupine2 hours ago|

parent|

[-]

At my day job I may get access to big AMD AI iron in a couple months (to do research/performance tuning with). That could be interesting. Though that's likely to be of a very different shape from consumer Vulkan. I'd still like to have a Strix Halo to futz with. But I'll wait for RAM prices to drop. (Hah!). I do have an older BC250 board lying around but that only has 16GB RAM.

I got prefill up to 190 tok/sec just now, BTW.