I have epyc 9654 ES and a 7900 XTX. I was running the numbers, and even if I maxxed out the ram to like 12x32 gig sticks, it would cost me thousands more and I could only run GLM-5.2 at a couple tokens per second at q3. So this project is very promising because it suggests I could get pretty high speed and this CPU/motherboard combination suggests I have a lot of pci bandwidth that is unused.

I think another route might be looking at holding an even larger chunk of model weights in ram, and taking advantage of RAM<->GPU bandwidth, perhaps using a PCIe 5 GPU. This was my first thought since I have dedicated GPU.

If you are using Laptop, you're looking at shared memory between the iGPU and CPU. I've also tried that route, but I have always been skeptical of killing flash with too many reads, it essentially uses SSD like it's a consumable item.

I'm going to benchmark this right now with what I have and I'll get back to you on github.

Good catch! Disk reads do generate writes to cache. But the cache itself is in RAM, not on disk. So it shouldn’t cause additional wear of SSD.

> Is this a hallucination? What am I missing? Why would heavy reads generate writes?

I take it heavy reads means more stuff goes into RAM, meaning other stuff has to be cached?

I've got same question as GP: e.g. is there a way to set moderately fast consumer NVMe SSDs (I've got both a Samsung 990 Pro and a WD SN850X) in a complete read-only mode to prevent "wear"?

Spilling

I also just edited my comment with more ideas in the beginning, sorry