undefined

points

[-]

Is there a particular quant of DS v4 Flash you'd recommend that works on 64GB machines? None of the antirez versions on HF look small enough?

Also, FWIW, I've been experimenting with Laguna-S-2.1. It runs reasonably quickly (llama.cpp, IQ2_M quant) but the outputs so far aren't impressive, and it gets stuck and perseverates. Very subjectively, at that level of quantisation, it seems to perform worse than Qwen 3.6 27B at Q4_K_XL.

by hacklas2 hours ago|

parent|

[-]

For a dense model this would be a limitation, but not all of a MoE model needs to be in memory, but the largest part of a MoE are the routed experts.

Some parts are needed to generated every single token and these really should fit in memory, but the router experts that are not neeed can rest on SSD and be read only if they are needed, so... you can run MoE models bigger than you memory, try the IQ2XXS.

It should work on your 64 GB after you enable SSD mode in DwarfStar (in MoEspesso it enables itself), while being slower, so... I am really hoping for good models between the 50-120 GB other than Laguna, there is a big gap right now unfortunately.

by mandeepj4 hours ago|

prev|

[-]

> 7–10 t/s

Maybe use it for overnight batch work! Hopefully, you aren’t suggesting it using for realtime conversations!