Prefill is gated by GPU compute, not CPU compute. A Epyc/Xeon CPU will never beat an M5 GPU in prefill. If you want something that can match M5 Ultra in prefill, you'd need to add beefy Nvidia GPU. However, the problem is that the GPU memory is separate from the 384GB system memory. And here lies that advantage of Apple Silicon. It's unified memory and accessible by the CPU and GPU.

So 3x more, plus the cost of a GPU (another 10k?). How is that value for money to get slightly better performance?

Excellent post. Heck yes. And with MRDIMMs coming, we're going to get another >50% boost in throughput per channel real soon, with a massive uptick in max capacity (4x).

It feels like PCIe is a bit of a boat anchor here. There's a SATA->NVMe style transition waiting in the wings to make this all so much better. We really need post-PCIe GPUs. CXL with it's very small low latency flits. This is an "almost certainly not" but I wonder if you could mix PCIe and CXL so you could have the GPU memory expose vmeme as a bunch of CXL.mem pools but still have an otherwise pretty normal GPU. It seems madness that UALink went all in on GPU-to-GPU with no affordances for connecting to host computers.