No, that’s a common misconception. MoE despite the name doesn’t have an “expert” slice that’s an expert in any particular thing. It’s just trained models such that a subset of all memory weights is active on a single token. Often, the weights active on the next token are different. It’s used to improve memory bandwidth (throughput) and also to even out load in a distributed serving model - one GPU hosts one “expert” and the conversation is routed to it for the next token. This requires a lot of movement of the KV cache of course so it’s difficult to retain speed across multi node splits - usually it’s a rack of GPUs that you do this for.

MoE's in the abstract often get presented as if theres explicit layers of experts for any given domain of knowledge, like your coding tasks are being routed to coding experts, but it's really not that at all.

THe original MoE paper from Noam Shazeer et al. is worth a read on this bit, though the paper is admittedly pretty dense. But TL;DR is that each expert layer is learning highly abstract, localized structural and syntactic patterns in the data to minimize the loss function, and its doing this token-by-token (which in some cases may have some domain clustering, but that's just incidental).

When you start batching your queries, even if they all seem like theyre in a single domain, if you visualized the activations you'd notice that most if not all of the network is lighting up on the batched forward pass.