> Apple doesn't design GPUs on-par with Nvidia's efficiency yet
replyHow much it matters in inference? Most GPUs have enough computing for that and the bottleneck is the RAM speed and size. And M5 Ultra is becoming to challenge this.
For prefill, it's basically all that matters. Long-horizon agent tasks, session compaction, file reads and context manipulation will all hit the compute bottleneck in regular usage, incurring several minutes of latency on most Apple Silicon chips, regardless of RAM.
replyIt's kinda why memory bandwidth is an enormous red herring, even for datacenter applications. Nvidia's huge advantage is a compute-optimized GPU architecture and their Infiniband networking, their memory controllers aren't really the star of the show.
Probably two generations away. I’m more interested in how much uplift/speed and more importantly what is the power usage is required for the new computers Apple is shipping particularly for the Studio versions.
replyApple has metal and mlx
reply