”Serve the model on either two Mac Studios, with M5 Ultra and 512GB of unified memory; or four Mac Studios with 256GB of unified memory, for a total of 1TB of unified memory, enough to hold a NVFP4 quantized version of Muse Spark Open”
replyCan not find any info on inference speed.
its around 20tps
replyRead it again. Two M5s.
replyWait read that again. Two 512GB, unreleased model M5s.
This is gonna cost like 50k to run at home.
Closer to ~$30K if I had to guess. A 256 version is $11,299.00 (2TB SSD, 256GB RAM, 36-core CPU, 80‑core GPU), so yeah, 4 of those would be close to $50K but the 96->256 RAM upgrade is $4K, if we assume 512 will be double that it's ~$15K/ea.
replyBut yes, still a massive amount of money and that's before we know the TPS or any other stats.
The closest equivalent to something like that is a GB300 based workstation, those go from 90K and up.
replyHow much does the engineers at Meta make? Spend enough time together with others making $250k+ and eventually you start suggesting that two 512 GB M5s is something you can put on your desktop. I mean, it's one banana, Michael. What could it cost? 10 dollars?
replyWhy would anyone want to run an LLM on Apple's hardware? Why not buy an NVIDIA GPU?
replyTry to find a NVIDIA GPU with 256/512 GB of RAM.
replyokay that's gonna be interesting to try
reply