LLM speed is roughly <memory_bandwidth> / <model_size> tok/s.
replyDDR4 tops out about 27Gbs
DDR5 can do around 40Gbs
So for 70B model at 8 bit quant, you will get around 0.3-0.5 tokens per second using RAM alone.
DRAM speeds is one thing, but you should also account for the data rate of the PCIe bus (and/or VRAM speed). But yes, holding it "lukewarm" in DRAM rather than on NVMe storage is obviously faster.
replyChannels matter a lot, quad channel ddr4 is going to beat ddr5 in dual channel most of the time.
replyFour channels of DDR4-3200 vs two channels of DDR5-6400 (four subchannels) should come out pretty close. I don't see any reason why the DDR4 configuration would be consistently faster; you might have more bank groups on DDR4, but I'm not sure that would outweigh other factors like the topology and bandwidth of the interconnects between the memory controller and the CPU cores.
replyyeah, actually, I'm bottlenecked af since my mobo got pcie3 only :(
replyFaster than the 0.2tok/s this approach manages
replyShould be active param size, not model size.
reply