I wonder when/if we will start to see Apple include a silicon encoded model into their chips. Similar to Taalas build Llama 3.1 silicon with 17,000 tok/s inference.
replySo could the M7 actually include an AFM 3B model, alongside a generic neural engine?
The M6 doubled the neural engine from 16 to 32 cores. I would expect that the M7 doubles that again to 64 from 32? That would make sense.
replyI believe that the CPUs are actually limited by ram bandwidth more than the neural engine right when it comes to LLM processing?
Maybe the M7 introduces something new to get around the current ram bandwidth problems on the non-Ultra chips.
Apple's biggest bottleneck for real-world inference is prefill processing. They need a better GPGPU architecture, which is what I'm expecting M7 to reveal.
replyAgreed, seems like the M5 has already made steps in that direction, with 4x prompt processing / prefill performance vs. M4. [1] And token generation also got a 10% boost. The data below is only for Pro & Max but I think the base M5 got the same relative boosts vs. M4 base.
reply Chip BW (GB/s) GPU Cores Q4_0 Prompt Q4_0 Gen
M4 Pro (20c) 273 20 439.78 50.74
M4 Max (40c) 546 40 885.68 83.06
M5 Pro (20c) 307 20 ~1500 to 1700 ~56
M5 Max (40c) 614 40 ~3000 to 3500 ~92
ELI5?
replyLook at the "Q4_0 Prompt" column. That's the tokens-per-second processing the initial prompt/system prompt. This is where the "neural accelerators inside each GPU core" is seen most prominently.
replyThe M5 GPU added better matmul and dot product support in hardware (IIRC) that really boosted performance of those kernels over previous GPUs. Current transformer models rely heavily on matmuls. Prefill (processing the current context) is mostly limited by raw GPU processing power where token generation (predicting the next token from the current context) is mostly limited by memory bandwidth.
replyWhile an older M-series might run a matmul or dot product kernel just fine the M5 can run them much faster.
Apple's biggest bottleneck for real-world inference is prefill processing.
LPDDR6 is coming.
replyI understand that will boost read rates to around 14 Gbps as compared to the current 10 Gbps for LPDDR5X, so a 40% improvement.
replyThe bus is also 50% wider so the bandwidth is double.
replyOr they could make use of LPDDR5X-PIM? That would be such a killer feature and competitive advantage.
replyreply
i sold my m3 ultra /512 for £14000, or about $18000. The difference is what is important to me for a new one, and assuming $6k or so for the 256->512 boost, my cost will be about $16k, so i’ll save $2k by upgrading to the top-of-the-range model, bar it being a 4TB drive
replyDo current models run on the NPU or GPU? Wondering if Apple will have something like a TPU.
replyApple has a dedicated "neural engine" which is designed as an inference NPU. Where as Google's TPU has a dual focus, both inference and training, which is a more complex design.
replyThat makes sense. I also think that we are at the point where maybe the expectation of a yearly refresh on everything should be reconsidered. They can end up on a Tick-Tock schedule when the Pro/Max/Ultra can be done every second year.
reply> I'd skip M5 and M6 chips for LLM work and wait for a year for M7.
replyPlease say more? Is it because it is a one-time cost, unlike a recurring subscription of Claude/Codex?
I'll upgrade M3 Air only when Mx Pro/Ultra can run Opus level perf locally. Otherwise what's the point.
replyThe true-local AI chip, codename "buddy", will be the M8
reply