upvote

points

by xyzzy12322 hours ago |
comments
upvoteby reliabilityguy19 hours ago|
next
[-]
> You could run MACs directly in RAM

Sure, MACs are nice. However, unless there other, PIM-specific/optimal, algorithms, regular matrix multiplication algorithms like tiling-based won’t work here I think — how would the tile be shared? By doing read/write all the time?

reply
upvoteby whatshisface11 hours ago|
parent|
[-]
Attention calculations aren't shared across more than one vector during next token prediction (thinking and writing) which this sounds almost perfect for. Per attention layer, for deepseek at 1M context, you want to broadcast a single 1KB vector to 4GB of dot products, and map reduce a 1KB vector back.
reply
upvoteby reliabilityguy10 hours ago|
parent|
[-]
How exactly the map-reduce will happen though? Won’t you need to do it host-side, or make a lot of reads and writes?

Also, doesn’t it mean that you forgo batching?

reply
upvoteby 9 hours ago|
parent|
next
[-]
deleted
reply
upvoteby ACCount378 hours ago|
parent|
prev|
[-]
Map-reduce is implemented as a rolling calc, see: online softmax in FlashAttention kernels.
reply
upvoteby reliabilityguy3 hours ago|
parent|
[-]
Rollie calculation, like the online softmax in FA, implies a centralized computing unit that does the compute and stores the intermediate results in its registers. With PIM you have no centralized compute unit, you have a bunch of memory, and a bunch of MACs all over the place.

How would you do map-reduce across multiple DIMMs w/o extra reads/writes?

PIM implies some sort of distributed compute, which can work for some cases, but I am not sure LLMs are one of them.

reply
upvoteby whatshisface1 hours ago|
parent|
[-]
Re-arrange x+(y+(z+w)) into (x+y)+(z+w) to accomplish a distributed sum. Shared bus, interconnect fabric. This is a supercomputer on a chip.
reply
upvoteby embedding-shape21 hours ago|
prev|
next
[-]
> Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.

Isn't this how it works today already? Granted you wanted to run it on RAM rather than VRAM.

reply
upvoteby petu20 hours ago|
parent|
[-]
Yes, but running out of RAM is impractical due to low memory bandwidth.

According to the article/Samsung RAM dies inside can support way higher bandwidth than they expose, they're limited by external interface / bus width:

> Together, they can utilize the chip’s internal bandwidth across all 16 banks, which comes out to 614 GB/s. For comparison, regular DRAM accesses can hit two banks in parallel and max out at 76.8 GB/s.

And that's just for single 64-bit IC. So way faster and more power efficient.

reply
upvoteby dannyw17 hours ago|
parent|
[-]
You can scale with more memory channels. Workstation/server platforms go up to 12 or 16 channels if I remember correctly.

Consumer platforms have been stuck at dual channel for decades; most of it I attribute to intentional product segmentation. I'm hoping that LLMs might change eventually for an upcoming consumer platforms; going to 4 channel would be really nice.

reply
upvoteby zerd54 minutes ago|
parent|
[-]
How do you scale past 16 channels though? 16 channels give you around 614 GB/s, while PIM can do that per chip, so it can achieve 58TB/s.
reply
upvoteby amelius20 hours ago|
prev|
next
[-]
You: "AI, please write me $COOL_APP."

AI: "Sorry, all the hardware is made for running AI."

reply
upvoteby eru18 hours ago|
parent|
[-]
We can run Doom on everything. Surely we can run some interesting apps on hardware that's originally made for AI. (One big moment for AI was when people figured out how to run it on hardware originally meant for Doom's successors.)
reply
upvoteby WithinReason20 hours ago|
prev|
[-]
[dead]
reply