> 3.8-27B announced for Friday
replyMaybe I’m misreading this or some other post, I thought QWEN was stepping away from releasing these models for local consumption
3.8-27B is confirmed for Friday. They didn't release their whole 4B-400B range of models since 3.5. And 3.6 only got 27B and 35B MoE. So yeah, slowing down, but not completely out of the small model game.
replyThey reversed course and now are saying they'll be releasing their Max style models in open weights.
replythank you china!
replyUnsloth already has a guide for their quants: https://unsloth.ai/docs/models/qwen3.8
replyI wonder who is unsloth and where they got time, hardware and knowledge to quantize them?
replyDaniel Han is just that good!
replyunshloth started as a finetuning library with lots of optimisations so you could finetune on lower end hardware. Kind of OGs of the local community. Started by two brothers Michael and Daniel(?) a math wiz and a community builder/communicator. They've since gotten some VC backing, are active in quantising lots of models on release day (work w/ labs to prepare things), known for their optimised quants (use different bits for different layers). Recently I saw they launched some sort of a desktop app, like lmstudio if you're familiar with it. They're really cool people and known in the local model places.
replyIt's our guy Daniel: https://www.linkedin.com/in/danielhanchen
replyThey started with offering training methods for quantized models to save memory and added new things over time. They are very active in the local model community and have extensive documentation and tooling to help with running and training models locally.
replyLlama.cpp can quantize without special training, but I'm not sure if any special model architecture support is needed to read it in the first place. If it can be converted to gguf at all and you know what tensors to target, it can get the full ternary bonsai treatment today.
replySure, but that's for "personal" serving. I meant for 3rd party providers. Usually we get a good indication on what it costs to host this, as the prices settle on open router. That's why I said it's tougher to serve than kimi k3 on launch. As a provider you'd do fp8 if the model creator didn't do QAT on q4, or until someone does a good calibrated nvfp4. And that's usually nvda :)
replyThat makes sense, but your specific phrasing precluded the possibility of non-QAT quantization.
replyShould have worded that better, my bad.
replyQAT is an optimizing quantization algorithm, not naive quant.
replyRight, but the way they phrased it suggested that without QAT it could not be quanted at all.
replyNow that they have reached the frontier in raw performance, I would like to see Chinese models improve their reasoning efficiency.
replyFor all the talk about over reasoning, K3 on low thinking has been rather nice
reply[flagged]
replyquanting is actually cheap and you can compress a model that does not fit on a GPU. You can process layer by layer, this is what the sequential processor in llm-compressor does.
reply