The framework desktop is quite cool, but those Ryzen Max CPUs are still a pretty poor competitor to Apple's chips if what you care about it running an LLM. Ryzen Max tops out at 256 GB/s of memory bandwidth, whereas an M4 Max can hit 560 GB/s of bandwidth.
replySo even if the model fits in the memory buffer on the Ryzen Max, you're still going to hit something like half the tokens/second just because the GPU will be sitting around waiting for data.
Personally, I'd rather have the Framework machine, but if running local LLMs is your main goal, the offerings from Apple are very compelling, even when you adjust for the higher price on the Apple machine.
There's also the DGX Spark. Granted, its price has been going up recently alongside everything else that has memory in it.
replyI haven't heard a single good think about DGX Spark from anyone using it, so I'd be pretty wary about that.
replyThat also has pretty poor memory bandwidth. 283GB/s I think.
replyYeah. The main selling point I'd say is the onboard ConnectX-7 hardware.
reply128gb is the max RAM that the current Strix Halo supports with ~250GB/s of bandwidth. The Mac Studio is 256GB max and ~900GB/s of memory bandwidth. They are in different categories of performance, even price-per-dollar is worse. (~$2700 for Framework Desktop vs $7500 for Mac Studio M3 Ultra)
reply128GB*
replyThanks for spotting the mistake. No Idea how I got to 192
replyFor what it's worth, I really wish that was the actual number.
reply