upvote
SOTA models are reportedly MoE, not dense.
reply