That's 160GB-ish for Q4...how is 256 insufficient?
replyLooks like the M5 Ultra Studio wait times are going to increase again. Already at 10-12 weeks, I wonder how long it'll go?
replyI guess like the M3 Ultra, at some point normal customers won’t be able to buy it.
replyThat M3 had an older type of RAM. Apple hopefully secured sufficient supply of the newer variant for the M5 Ultra.
replyBoth use LPDDR5x, they're not shipping LPDDR6 (yet).
replySpeaking as someone who isn't really well versed in this, does 18B active parameters mean that you could potentially hold only the 18B parameters in RAM and stream the rest from a fast NVMe SSD for acceptable performance similar to how Colibri works?
replyNormal MoE is switch-weights-per-token so you would se substantial slowdowns that way. Apple did a More that switches weights per prompt (instruction-following pruning, https://arxiv.org/abs/2501.02086) but you have to design the model that way which I don't think the have.
replySorta, but you're off by one layer. You can store the 18B in VRAM and stream the rest from RAM. There's still a performance hit relative to storing it all in VRAM, but it's tolerable.
replyGenerally, for local consumer use, these large MOE models are best for unified RAM systems like DGX Spark or Mac Studio.