That's kind of the problem, isn't it? How do you know which part of the model to put in memory? You have to make a per-parameter decision of whether or not it's worth it to have it in memory or whether the value should just be treated as zero. Then you have to "re-link" the layers of the model to the new positions of each of the weights. For billions of parameters, that's a lot of calculations. And it requires us to know what each parameter actually represents, which nobody does.

that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

The A in 26B-A4B is the active weights.

The problem is that this is a per-token load/unload at best, not for the whole prompt.

The division happened until one of these can fit in a single GPU and they stopped scaling it down any more, because you can wire up 8 of them to do their share of the work.

Always curious when someone will figure out how we can elide most of the data from an LLM (but retain the logical ability). I don't actually need an LLM to have a very big internal knowledge base to be useful, so long as it can invoke a search tool...

It's still early days and we "just" don't really know how to do it well.

Dense LLMs typically perform better, but slow down much more than MoE models when you try offloading layers.