Yes, the "neural accelerators" in the M5 GPU cores really do 4x pre-fill performance over M4. You can find benchmarks online since M5 has been out for months now. I assume the M6 has whatever the next-gen version of those is. These are not the "neural processors" that have been there since M1 (those still exist) but are additionl matrix math accelerators inside the actual GPU cores. Like "tensor cores" on Nvidia GPUs.
replyThank you, found some benchmarks and they look really promising. To my understanding those Neural Accelerators are like AMX but for GPUs. With those accelerators the GPU performance on a M5 Max in LLM inferencing would totally be on par with a 5090, that's quite impressive!
replyWhich inference benchmark are you looking at? The prefill speeds should be comparable on some LLMs, but the 5090 has much a higher theoretical max decode speed.
replyI found some on Reddit, and here are the two benchmark results from review sites:
- against a desktop 5090: https://nanoreview.net/en/gpu-compare/geforce-rtx-5090-vs-ap...
- against a laptop 5090: https://nanoreview.net/en/gpu-compare/geforce-rtx-5090-mobil...
replyWith larger (also really fast) unified memory on the chip, it could easily load larger weights, and the thing is that the quantized results on the M5s are really good, I believe that for most local inferencing, users would be using INT4 (maybe more bits per weight sometimes) quantized models, that might be where the Neural Accelerators kick in. In raw power, a desktop 5090 easily outperforms the M5 Max, but for this specific use case, I believe that the M5 Max is good enough.