I run QFN I2 XS on my 3060ti + 48GB RAM for it. Get around 2x token with 200k Context window. Good enough for me.
replyErr strata is just a fork of llama cpp with fancier uis, right?
replyStrata manages MoE expert offloading completely differently, which for QFN means ludicrous speed.
reply