yeah, actually I wanted to see if this was possible at all. I managed to get around 3000 tokens/s on a ps2 with classic transformers, since the emotion engine is capable of 32 bit addresses, but it has like 32gb of ram. So I ran into the question of why was that fast and I couldn't get that speed even with small models, and the deal is that the instructions went right of the memory to the gpu and that's the main difference that does when a regular computer does inference: it has to request the instructions to the cpu every time. As I mentioned too, on professional cards you can avoid these problems naturally, since they got instructions precisely for this, but sadly I don't have 30k bucks to spare on a gpu :(
reply*32MB of RAM (plus 4MB of video RAM and a little sound and IOP memory).
reply> I don't have 30k bucks to spare on a gpu :(
replyDo you have $2/hr to rent an RTX 6000 96GB or $5/hr for B200 180GB on the cloud?
I thought about that, but idk if they allow me to modify the linux kernel and nvidia cuda kernel at all
replyIn those systems you could probably leverage something like Nvidia SCADA or GDS directly.
replyActually since they have direct GDS it should perform really well on professional gpus
replyI think you can do a bunch of that on Digitalocean's GPU droplets.
replyI'd rather not give money to scalper barons if I can avoid it. Fab capacity is going to that for rental rather than hardware for humans.
reply3000 tokens per sec on 32 mb Ram?
replyfast != practical
replyYou can get lots of tokens per second on the CPU if the entire network fits in L1 cache. Unfortunately the sub 64 kiB model segment isn't looking so hot.
But actually ... 3000? Did GP misplace one or two zeros there?
I wondered the same, but the rendering seems right, the output was almost instant. I'll recheck the token counter; anyway as you say, fast isn't practical. Actually I had to develop my own tiny model https://huggingface.co/xaskasdf/brandon-tiny-10m-instruct to fit something "usable", and it's basically a liar or disinformation machine haha
replyI can imagine a couple scenarios in which a high-quality, large model would be much preferred over lower latency models, primarily when you need the quality.
replyI didn't really understand the performance table until I saw the top ones were 8B models.
replyBut 5 seconds / token is quite slow yeah. I guess this is for low ram machines? I'm pretty sure my 5950x with 128 gb ram can run this faster on the CPU with some layers / prefill on the 3060 gpu I have.
I also see that they claim the process is compute bound at 2 seconds/token, but that doesn't seem correct with a 3090?
LLM speed is roughly <memory_bandwidth> / <model_size> tok/s.
replyDDR4 tops out about 27Gbs
DDR5 can do around 40Gbs
So for 70B model at 8 bit quant, you will get around 0.3-0.5 tokens per second using RAM alone.
DRAM speeds is one thing, but you should also account for the data rate of the PCIe bus (and/or VRAM speed). But yes, holding it "lukewarm" in DRAM rather than on NVMe storage is obviously faster.
replyChannels matter a lot, quad channel ddr4 is going to beat ddr5 in dual channel most of the time.
replyFour channels of DDR4-3200 vs two channels of DDR5-6400 (four subchannels) should come out pretty close. I don't see any reason why the DDR4 configuration would be consistently faster; you might have more bank groups on DDR4, but I'm not sure that would outweigh other factors like the topology and bandwidth of the interconnects between the memory controller and the CPU cores.
replyyeah, actually, I'm bottlenecked af since my mobo got pcie3 only :(
replyFaster than the 0.2tok/s this approach manages
replyShould be active param size, not model size.
replyThat's slower than just running it off CPU+GPU. I can easily hit 1.5 tokens/s on a 7950X+3090 and a 20480-token context.
reply