For single stream inference of a MoE model, the size of active sparse parameters will matter a lot more than total parameters. This is generally around half of the reported active parameter count - the other half being a dense subset that can be easily cached in VRAM even on fairly modest consumer setups. So the achievable performance may be quite a bit better than a naïve assessment might suggest.
replyOn a server machine you can have more than 100GB/s of NVMe if you parallelize (RAID 0 and the like). But it's still gonna be noticeably slower.
reply1536GB of DDR4 ECC server RAM is somewhere between $4000-6000 USD used right now, by the time you put in parallel enough NVME SSD to approach good speeds, you'd be approaching that (and also likely running out of PCI-E bus lanes directly attached to the same motherboard to reasonably do so).
replyIt claims to support using multiple devices RAID-0 style, which should boost performance, but yea probably not very useful for most.
replyBut still fun you can run it at home.