upvote
> You have to look at the size of each expert

Yes, this part is accurate. Expert density determines how much raw compute each hidden state gets.

> The number of the experts tells you about the diversity of its skills.

Most people misunderstand this part. Counter-intuitively experts don't develop diverse skills, they instead balance compute during the forward pass, allowing models to increase their parameter count without the MLP layers exploding in memory + compute requirements.

reply