undefined

points

[-]

DeepSeek-V4 should use only 5GB for context due to CSA and HCA, see figure here: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro

But not every framework implements it properly yet.

by walrus0110 hours ago|

parent|

[-]

Yeah, or close enough to 5GB for estimation purposes, for example a just spawned qwen 3.5 122B llama-server instance reports as:

0.07.015.888 I common_memory_breakdown_print: | - Host | 170038 = 162913 + 6740 + 384 |

The 6740 is the cache size.

by Iolaum11 hours ago|

prev|

[-]

There's an emerging practice of using Q4 quants and Q8 KV cache for local inference. At that point you can run both Qwen3.5-122B-A10B (my personal choice on Framework Desktop 128gb) and Laguna-S-2.1.

Now whether that's good enough for one's use-case remains to be determined. You can also get more out of those (local models and quantizations) if you further tweak the harness you use them with, but tbh this is where it gets too much work (at least for me and the time I have available).

by zargon10 hours ago|

parent|

[-]

> emerging practice of using Q4 quants and Q8 KV cache for local inference

That's not an emerging practice, it's a tested strategy that is these days only used as a last resort by those desperate to fit a model in memory. Some models do better than others, but generally the model quality suffers greatly under those conditions.

by walrus0110 hours ago|

parent|

[-]

I have never seen anyone report "this produced really great results" from intentionally quantizing their context vs. leaving it at full precision which is the ordinary default.

by trollbridge7 hours ago|

parent|

[-]

Gemma's QAT is surprisingly good (although Gemma isn't that great to begin with).

by KerrAvon6 hours ago|

parent|

[-]

IME: Gemma is not great for programming, but it is fantastic at following directions compared to anything else in its size class.

by apitman6 hours ago|

parent|

prev|

[-]

Artificial Analysis ranks qwen3.6-27b higher than qwen3.5-122b-a10b on both intelligence and coding. Does that run counter to your experience?

by nekitamo3 hours ago|

parent|

[-]

This tracks, in my experience the 27B is better at coding and instruction following. I'm shocked at how much of a difference the dense models vs MoE makes.

But it's a moot point, because for local inference on consumer hardware, the MoE is so much faster.

by apitman3 hours ago|

parent|

[-]

Are you sure the difference is from MoE and not that 3.6 is newer?

by riknos3142 hours ago|

parent|

[-]

Qwen 3.5 27B also scores higher than 3.5 122BA10B. So even in the same generation the smaller dense model outperformed the larger MOE