> if someone would kindly quantise this down for the 64GB paupers
replyThe Q4_K_M is 75GB. I'm exactly at 64GB, and I wouldn't quantize it further. Instead, do partial weight residency and stream the rest from SSD.
They have also published smaller 33B model called Laguna XS 2.1, its Q4 gguf is 20GB.
replyhttps://huggingface.co/poolside/Laguna-XS-2.1-GGUF/tree/main
Thanks for flagging. From the few benchmarks I can find, it looks there or thereabouts with Qwen 3.6-35B-A3B, or maybe a touch below. I'm interested to compare a model that is a big jump larger with pretty impressive benchmarks, but more heavily quantized to fit.
replyThe tool I've been using, llm-compressor, can quant models that do not fit in memory (use the sequential pipeline)
replyhttps://github.com/vllm-project/llm-compressor
my setup to help you on your way: https://github.com/verdverm/quantr
Though it seems these will not be needed as Poolside has published quants & dflash with their models.