I tried some 1-bit, 2-bit, and bonsai quants against closed eval sets. They were essentially useless for my case. The little errors accumulate and send the whole output off track quickly.
replyIf you had some use case with very small output sequences they could be interesting to try. I think dropping down to a 9B-class model would produce better results for most cases.
I wonder if this would help, or if it solves different kinds of errors.
replyShow HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
What setup are you using to do said private evaluation? Software wise I mean
replyNot 1-bit, but I’m getting pretty good results with some light coding using unsloth’s previous 2-bit quant of qwen3.8-27b. With these new quants i may be able to bump up to 3bit, tho it’s already running so slow (15tok/s average for the first 32k of context) that the speed hit might make it not worth the extra smarts
reply