Sharding a dense model using Tensor Parallelism (TP) across dual RTX 5090s has a significantly worse performance penalty over PCIe than sharding an MoE model.

If you are using multiple GPUs, MoE is basically going to be your only workable choice unless you can leverage pipeline parallelism (only half your GPUs can work on a prompt at a time, so you need to process prompts back to back in a pipeline setup, and they better be doing similar things because your vram is limited).