Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective.
replyRather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.
Baking the base models on to ROM makes a lot of economic sense.
replyLess so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.
And the customers can wait for the new phone released next year. These are edge models - the average customer doesn’t need the latest frontier model. Just needs to be good enough for the features you promised.
replyIt's a perfect reason to get consumers to buy a new phone every year again! They got bored of the camera.
replyIt is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.
replyThe big benefit is ROM cells require fewer components than DRAM. So the chips would be tiny, dense, cheap and consume far less power.
replyI thought DRAM was pretty dense already. Is mask ROM that much denser?
replyYes, each rom bit can be a transistor or even a diode with a decoder circuit. Simplest Dram cell is capacitor+transistor - and you need a clock, refresh circuit etc.
replySomeday, I imagine model weights could even be encoded as analog resistors (memristors or similar) for even greater density
Hm. I wonder how many relays I'd need to make a physical MNIST classifier. That'd be dope
replyTheir PoC chips are big, but then it's ridiculously fast (have you seen chatjimmy.ai?). Also they must be holding a bunch of patents.
reply