What you're describing is what Cerberas does.
replyTalaas is different, it's a true compute-in-memory architecture where the weights are stored in the connections between the transistors that perform the matrix multiply, rather than in seperate memory cells.
Most of the benefit comes from this architecture; hardwiring the weights into the silicon is just the easiest way to implement it. SRAM requires too many transistors, DRAM requires an incompatible manufacturing process, and exotic phase-change memories aren't readily available.
Cerebras chips are massive and do have more on the edge but they dont have any top or bottom cache do they?
replyThey can't due to power density, I believe - they have to be run in a sandwiched waterblock with massive cooling, as far as I can tell. That's the biggest thing that baked weights gets you - a relatively modest watts-per-square-mm compare to cerebras, where they had to engineer a whole system to get the watts out of the chip
replyDo you think there's room for reducing power requirements? Obviously shrinking the process is a win, but is the existing implementation a "just make it work" phase that has opportunities to increase computational efficiency?
replyWith that kind of speed and if even lower power requirements, they could release mini compute units with USB4/Thunderbolt for plug and play inference.
I mean I think I read their spec sheet that it was 2.5kw on the asic but closer to 25kw with GPUs, for the same number of tokens (different latencies of course), but there's no way that a dedicated from-scratch matmul+memory chip can be less efficient than a GPU.
replyTaalas does not have cache so...
replyI agree that Groq with multilayer hybrid bonding could be a good idea.