My experience also aligns with this. I'm running gemma4 31B on a 4090 through llm.cpp with unsloth models.
I also run Qwen 3.6. Qwen is good for thinking and planning as it is faster, but Gemma4's generated code is much higher quality in the first try (Rust, C++ and C#). so it needs less revisions to be at a level I'm comfortable for merging.
replyI second unsloth models. I'm using them over blackwell-oriented nvfp4 models as they are (empirically) top quality and performance.
replyNVFP4 will be better if the model provider actually post-trained properly after quantizing.
replyWhich basically only Nvidia does, because it’s very expensive.
replyThough I’m currently working on QADing the smaller Qwen 3.5 models from FP16 teacher to NVFP4 student, to hopefully eventually apply it to 3.6 27B… harder to get right than I expected though!
deleted
replyI can't Gemma4 to actually finish a turn properly, it's always ending abruptly or making malformed tool calls. It's probably something I've misconfigured in oMLX or Opencode.
replySame problem with Gemma 4 + oMLX + OpenCode. The thinking and tool calling seems to be parsed fine in other clients such as Open WebUI. This really shouldn’t even matter because the client isn’t responsible for parsing the output, but it’s happening anyway.
replypossibly a problem with the chat template
replyhttps://huggingface.co/google/gemma-4-31B-it/discussions/118
Huh. Same problem, and I run with llama.cpp. In my case, Gemma4-31B (4-bit quant though) will just stop sometimes.
replyNice. I flip flop between Qwen 3.5 9B Q6_M and Gemma4 12B Q4_K_M on a 4080 Super. They run at about the same speed and I can have them review each other's plan or diffs. For smaller projects I find them very capable, and I can step up to a better quant for slightly more challenging work.
replyyou can probably run Gemma4 26B on your card also at 4 bit. World of a difference compared with 12B.
replyWhere does “big model highly quantized” start getting worse than “smaller model less quantized”? Is there a general formula or is it just trial and error?
replypaper is a bit old, but matches current empirical recommandation: a good starting point is the biggest model you can fit at 4 bit
replyHave you tried qwen 27b q4_K_XL? It's a little bigger than the 4080 but not too much
reply