Not for me. As stated elsewhere, even Q5 (!) seems to be enough to kill the model’s ability to solve a particular problem in reasonable time. But that might just be right at the edge of what the model can do in the first place.
replyI have another personal benchmark problem (of a very different nature) that Qwen3.8-27B usually can’t solve at all, while Opus and GLM-5.3-Flash solve it consistently and very beautifully.
As they say, time is money.
replyIn the age of the rampocalypse, the peasants may not have a choice between the two.. time it is!
Smaller models are also generally faster, so thinking "more" may not matter and may even come out ahead.
replyIf Q4 takes less than 1.3x as many tokens as bf16 or q8, it could still end up being faster, given how decode tends to be bandwidth bound.
The kv cache was still bf16, so a few ops are the same between quants.
replyComputer science has known the tradeoffs between memory and compute since ages ago. The same could be reflected here.
replyI personally think thinking is basically variable but rate precision. If you are in a 4bit mode but need 2x as many tokens you're just doing fp8 with hoops( of course 4bit multiply is faster)
replyFair enough mental model, except my GPU can’t load the 8bit version and paging from disk makes it way more than 1/2 speed.
replyIf it digs itself into a hole, try low or medium. In the rust coding benchmarks (on my machine) it did better on low and medium because xhigh never finished.
replyI observed this as well. I have problems that get solved very quickly using non-thinking mode, but in thinking mode it digs itself into giant holes, that it sometimes doesn't manage to escape from.
replyLike wanting to implement an entire emulator to figure out something it could easily glean otherwise (and does, in non-thinking mode).