upvote
Rather than waiting, ask your assistant to quantize for you. It could be not the best quantization possible but it’s relatively quick, and I had an overall positive experience.
reply
It is already quantized. You need to convert to GGUF format though. Did your assistant explain this wrong?
reply
My bad, I haven’t really looked yet (and haven’t asked any assistants), thought it’s BF16 safetensors as usual, and for those I used to quantize to ggufs with just a couple commands.
reply
It isn't if you train the quantized model with a quantized K/V and cache. That's my approach and usually takes around a day for 100k reasoning traces (including abliteration).

Also this is on consumer hardware, meaning 96GB CPU RAM and 48GB GPU VRAM, using an optimized Vulkan concurrent shader pipeline. So costs for the machine are around 3-5k depending on RAM crisis.

[1] https://github.com/cookiengineer/qwen-reasoning-trainer

reply