If you just want to run Qwen 3.8 27B and Deepseek v4 Flash in perpetuity and that's it, there are a lot of solutions that will work and this is a fairly user friendly one.
Lastly, I want to clarify that prefill on an x86 CPU is drastically slower than on an M5 Ultra GPU.
It feels like PCIe is a bit of a boat anchor here. There's a SATA->NVMe style transition waiting in the wings to make this all so much better. We really need post-PCIe GPUs. CXL with it's very small low latency flits. This is an "almost certainly not" but I wonder if you could mix PCIe and CXL so you could have the GPU memory expose vmeme as a bunch of CXL.mem pools but still have an otherwise pretty normal GPU. It seems madness that UALink went all in on GPU-to-GPU with no affordances for connecting to host computers.
Good for inference; however if you like to train, data format support and effective performance is limited (M5 Pro). Some hardware features are not exposed or extremely slow.
You’ll be fine for inference, but pales in comparison to what a RTX 6000 Pro can do for compute/matmuls/training.
RTX 6000 wins in performance, if your model can fit into the VRAM.
There are very obvious and clear advantages to a Mac Studio. It's an entire system for one and you're getting a world class CPU as well.
[citation needed]. I have personally specced out and built an nvidia GPU-based machine which after some optimization, handily beat the Mac Studio in terms of tokens/watt for LLM inference with most models. This was in the M2 Ultra era, and I haven't run the numbers for the later generations, but nvidia's cards have gotten faster just as Apple's CPUs/GPUs have, so I would guess that it's still possible to do.
> RTX 6000 wins in performance, if your model can fit into the VRAM.
"if your model can fit into the VRAM" can be true for the Mac as well.
> There are very obvious and clear advantages to a Mac Studio.
There are certain advantages for sure, depending on your use case. They may _seem_ to be obvious, but as evidenced above, I believe that many people overestimate the Mac's superiority on the metrics you cite when comparing a Mac vs. a dedicated GPU for LLM inference.
It is much more likely for your model to fit in large unified memory of a Mac than the smaller more limited memory of a GPU. Even going with two 5090s, you now have to shard your model and that is a PITA.
But it turns out that MoE is the solution both for running models on macs of limited computer power means (not as fast as GPUs), and on multiple GPUs that require sharding the model.
I bristle at general statements like this when it obviously depends on the specific Mac and GPU in question. But yes, comparing a maxed out M5 Ultra with an RTX 6000, the Mac has much more memory.
> Even going with two 5090s, you now have to shard your model and that is a PITA.
Every modern tool does this for you automatically. It is absolutely not a pain in the least (e.g. llama.cpp ships with pipeline parallelism enabled by default).
If you are using multiple GPUs, MoE is basically going to be your only workable choice unless you can leverage pipeline parallelism (only half your GPUs can work on a prompt at a time, so you need to process prompts back to back in a pipeline setup, and they better be doing similar things because your vram is limited).
[citation needed].
RTX 6000 wins in performance, if your model can fit into the VRAM.
I can run agents using deepseek v4 flash or Qwen 3.8 on my m3 ultra and it will be lukewarm and the fan will eventually start blowing softly.
Yes, the Mac might get lukewarm, but it will take 2-3+ times longer to do the same task.
If you can show me a model for which a Mac is faster than the RTX 6000 then I’ll be happy to update or retract my statement.