I run this exact setup with 24GB vram and 288gb ran. I get 25 to 28tps. The catches you have to run the 2.25-bit version currently. Strata is not optimized to run the 3 ir 4-but versions, at least not yet.
With a 256K context window and vision I think it uses around 135 MB— I can’t remember if that’s total or just ram.
Edit: 4.1 flash next
reply> Strata is not optimized to run the 3 [o]r 4-b[i]t versions, at least not yet.
replyHow so? This is not obvious to me; AIUI Strata does support streaming weights for larger models via SSD. Of course the speed would drop quite a bit once you rely on SSD, but it ought to be workable. Since you're on a HEDT/server platform, you can also benefit by adding more SSD hardware to be accessed in parallel via the abundant PCIe lanes.