undefined

points

[-]

Maybe is a bit more niche, but I'm pretty sure there is plenty of people who have enough RAM to run dense models like this one, I don't know how slow is slow but people are willing to sacrifice speed if it means they get a good model running locally.

Look at "SC117/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-GGUF" on HuggingFace, It says it has 3.5M downloads this month and its offering a model that is 2-bit and 3-bit quantized (66GB - 83.9GB).

If this gets quantized to 4-bit (Q4_0) which usually there is very little loss in intelligence compared to the base model, then the model will probably be around 40GB, and looking that people are downloading a model that is at least 20GB bigger, then it might allow even more people to run a bigger model.

But it will only be popular among hobbyist if its as good as it claims to be.

by sorenjan12 hours ago|

parent|

[-]

Qwen 3.8 Flash Next is a mixture of experts model though. It has 6B active parameters per token, so you can cache large parts of the model in RAM and move it in and out of VRAM as needed. If you do that with a dense model it gets much slower.

by Tade011 hours ago|

parent|

[-]

Experts are not typically moved between VRAM and system RAM during inference, as it would be too slow.

What happens instead is that the "dense" parts of the model like e.g. the router are put into VRAM and the remainder of VRAM is occupied by the KV cache and whatever experts still fit.

The rest - usually just the experts - reside in RAM, but it's bo big deal.

Additionally, some models utilise an n-gram table, of which only one row is read per token, so it can actually be stored on disk without issue. Example:

https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC...

by petu6 hours ago|

parent|

[-]

> and the remainder of VRAM is occupied by the KV cache and whatever experts still fit.

Those days engines try to optimize "whatever experts" part and can use remaining VRAM as cache, moving hot experts to GPU during inference (but less used experts keep executing on CPU, yea).

e.g. llama.cpp recently added flag for that: https://github.com/ggml-org/llama.cpp/pull/29887

by Tade02 hours ago|

parent|

[-]

I was not aware of that, so I downloaded the latest release (which includes but doesn't document this flag), but performance unfortunately tanked in my case.

Even If I set the fit target to a value which without the cache would leave that VRAM unused, I get about a third of my usual token rate.

by entrope10 hours ago|

parent|

prev|

[-]

> Experts are not typically moved between VRAM and system RAM during inference, as it would be too slow.

For some definition of "typical", sure, but it depends on what performance trade someone is willing to accept. Strata LLM is geared around streaming weights as needed. It lets me run DeepSeek-4.1-Flash 1031, heavily quantized (ISTA-DASLab's 2.25 bpw quant), on a single Radeon R9700 plus 256 GB host RAM. 500 tps prefill and 30 tps decode isn't great, but it's a large fraction of what I get with Qwen3.8-27B quantized to fit entirely in VRAM for short prompts, it's faster past about 80k context, and DS4.1F can use a 1M-token context. It takes forever to fill that, or even to refill if the KV cache doesn't match, but it works.

by Tade01 hours ago|

parent|

[-]

I've tried llama.cpp's new --moe-cache-mib flag and it lowers performance to about a third of what it is without the flag.

And no wonder really - PCIe 4.0 x16 has a one direction bandwidth of around 31.5GB/s - much slower than system RAM - in my case about a third the speed and I think it's no coincidence that I'm getting this exact rate of slowdown.

by julianlam10 hours ago|

parent|

prev|

[-]

Something doesn't seem right. Vanilla llama.cpp can hit 50 t/s on an R9700. Q6 (albeit with limited context of 64k, more if kv cache is quantized) fits entirely on the card.

by stewartpark12 hours ago|

parent|

prev|

[-]

I hear what you're saying - there are people who will use it even at 0.1 tokens/s and run tasks over night or over multiple days if it is good. but Qwen3.8 Flash Next is a MoE model (6B active) so the memory bandwidth/compute required to calculate each token is very different from a 65B (meaning, 65B all active) dense model!

by deepsquirrelnet10 hours ago|

prev|

[-]

> it's a 65B dense model and inference will be prohibitively slow on consumer grade hardware

Times are changing and with MTP and DFLASH, dense models are back in style. For qwen 3.8 27b, you can get great token generation rates with MTP and for quality, it's much better than a MoE of comparable size.

by stewartpark9 hours ago|

parent|

[-]

That helps, yes. But speculative decoding that requires a secondary/draft model means a trade off (smaller KV pool), so often times the only option left is MTP unless you have lots of VRAM. And as you already know, these options (MTP, DFlash, DSpark, etc) are also available for MoE models, too. If it's 3tk/s instead of 1tk/s, we can say it helps but not sure how many people would be excited about that. I'd be more excited about 150tk/s instead of 50tk/s!

by 59nadir14 hours ago|

prev|

[-]

Is it competitive overall or just competitive when compared to "U.S. Open-Weight models that are otherwise pretty decent at agentic coding"? The latter does not imply the former.

by regularfry14 hours ago|

prev|

[-]

What'll be interesting is how it performs relative to Qwen3.8 27b when quantised to the same size. Which, of course, is a comparison they don't want to do.

by 40four14 hours ago|

parent|

[-]

Well, obviously. These newer American models aren’t competing with Qwen yet. They are competing for American businesses and organizations that can’t/won’t use Chinese models.

They are playing catch-up, but I’m happy to see new players to the game, and it will be interesting to watch how they develop