> Unfortunately that is nowhere close to being able to run a 750B param model. For something like that, we're getting closer to 1TB VRAM
replyYou don't have to run a model from VRAM, or even from a sizeable amount of RAM. These choices only ever make sense when serving the model at scale, to hundreds of simultaneous users or more.
For workstation inference a unified memory architecture would be a good cost/performance balance, while keeping COGs reasonable.
reply512GB unified memory macs are available, with the ram upgrade costing a few grand.