How much VRAM total you using for this? I have a bunch of 3060s in a threadripper and thinking I might need to give Flash Next a try. Currently using 27B.
reply64GB across 2 cards. You need the MoE caching build and enough regular RAM to fit everything else to get decent performance. You should get reasonable performance if you have enough RAM. I have a 4080 that does around 40 t/s right now on a MoE caching build. It's great. Not fast, but chugs along.
reply