If I had 128GB unified RAM I'd try hf.co/unsloth/Qwen3.8-27B-GGUF:BF16 which needs 55GB for just the weights.
replySomething like this would give you three concurrent sessions, each with 240k token context:
sudo sysctl iogpu.wired_limit_mb=110000
llama-server -hf unsloth/Qwen3.8-27B-GGUF:BF16 -c 786432 --parallel 3 -ngl 99 -fa onGreat news. Qwen 3.8 Flash Next (125B A6B) is coming out tomorrow. 4 or 6-bit should run nicely on 128GB.
replyShould bench better than Opus 4.7.
I have had a difficult time with running 120b models on my 128gb setup, especially with any larger context size. The 6bit of Qwen 3.5 is already just over 100gb, and when you go down to 4bit it seems a bit lobotomized.
replyIt's not exactly straight out of one of the labs -- it's heavily quantized -- but have a look at [1].
replydeleted
replyWell the upside is that you can run a laptop-sized model and still have enough memory left over to run a couple of Electron apps.
reply