> emerging practice of using Q4 quants and Q8 KV cache for local inference
replyThat's not an emerging practice, it's a tested strategy that is these days only used as a last resort by those desperate to fit a model in memory. Some models do better than others, but generally the model quality suffers greatly under those conditions.
I have never seen anyone report "this produced really great results" from intentionally quantizing their context vs. leaving it at full precision which is the ordinary default.
replyGemma's QAT is surprisingly good (although Gemma isn't that great to begin with).
replyIME: Gemma is not great for programming, but it is fantastic at following directions compared to anything else in its size class.
replyArtificial Analysis ranks qwen3.6-27b higher than qwen3.5-122b-a10b on both intelligence and coding. Does that run counter to your experience?
replyThis tracks, in my experience the 27B is better at coding and instruction following. I'm shocked at how much of a difference the dense models vs MoE makes.
replyBut it's a moot point, because for local inference on consumer hardware, the MoE is so much faster.
Are you sure the difference is from MoE and not that 3.6 is newer?
replyQwen 3.5 27B also scores higher than 3.5 122BA10B. So even in the same generation the smaller dense model outperformed the larger MOE
reply