> Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it.
replyHave you tried it with MTPLX? I get around 30 tok/s with it, also on an M1 Max with 64GB.
Even at 30 t/s, 3.8 thinks so long, even on medium, it still takes 3x or more longer than any cloud model, in my testing.
replyI've got an M1 Max 64GB too. It's just not an LLM-class workstation. Give it a year and buy an M7 and you'll be laughing. Right now is a really bad time to invest in anything - using the cloud is the cheapest option, especially for open weight models.
replyNice, which model quantization is this? Is it on huggingface?
replyMTPLX is this software: https://github.com/youssofal/MTPLX
replyI tried it with the author’s 4-bit quant of Qwen 3.8 27B: https://huggingface.co/Youssofal/Qwen3.8-27B-MTPLX-Optimized... (but no need to download it manually; MTPLX will ask which one you want).
Thanks, man! I’ll go use that now that I know. llama-server the last time I used it for inference with this model wasn’t able to produce work fast enough to reach those numbers.
replyI leave it running at night. No danger of burning my token subscriptions and it has hours and hours to run slowly with a manager like: github.com/kunchenguid/gnhf
replyHave you tried https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit ? PrismML is the only people i am aware of doing 1bit that is decent.
replyI've ran some agentic stuff with Qwen3.8-27B-UD-Q4_K_M on my RTX 3090. It's fast enough to be usable in my opinion. But Qwen3.6-35B-A3B in the same quantization is much faster
replyI treat it like image gen. Send a prompt then come back in 40 minutes.
reply