> Their stuff is already better than what nVidia is offering with stuff like the DGX Spark.
replyHow so? In tokens per second when running major open-weights models, or something else?
Apple isn’t the company that eats margins but they are company that would design around the problem and I think that’s what they will do after all, they have the design and engineering and plenty of money because they didn’t burn it on AI models or data centers.
replyApple doesn't design GPUs on-par with Nvidia's efficiency yet. They need an architectural overhaul to be a serious competitor, which is what I'm expecting is queued up for M7.
replyNvidia has CUDA, AMD has CDNA, and Apple has... compute shaders, I guess?
Probably two generations away. I’m more interested in how much uplift/speed and more importantly what is the power usage is required for the new computers Apple is shipping particularly for the Studio versions.
reply> Apple doesn't design GPUs on-par with Nvidia's efficiency yet
replyHow much it matters in inference? Most GPUs have enough computing for that and the bottleneck is the RAM speed and size. And M5 Ultra is becoming to challenge this.
For prefill, it's basically all that matters. Long-horizon agent tasks, session compaction, file reads and context manipulation will all hit the compute bottleneck in regular usage, incurring several minutes of latency on most Apple Silicon chips, regardless of RAM.
replyIt's kinda why memory bandwidth is an enormous red herring, even for datacenter applications. Nvidia's huge advantage is a compute-optimized GPU architecture and their Infiniband networking, their memory controllers aren't really the star of the show.
Apple has metal and mlx
reply