AI inference is basically 1 operation iterated over every single model weight. That's what's makes computer in memory attractive and feasible.
replyGeneral purpose workloads would require much more silicon real estate and are not (typically) nearly as bandwidth hungry. Just like not all CPU workloads benefit from being adapted to a GPU, even fewer will benefit from being adapted to processing in RAM.
AI is not intrinsically matmul. Current models are a framing of the problem as matmul to enable the bulk operations necessary. There are many calculations that can (and have been) reframed into matrix form to achieve similar results. More than once I have seen a matmul version of something that I thought couldn't have been representable that way. I'm not sure if the greater benefit is the performance or the ability to conceptually manage the problem in you head.
replyNot all algorithms can be converted of course, but a lot of that is because there are some inherently unparallelizable problems.
Many algorithms can be reasonably converted into matmul. Not many algorithms can be reasonably converted into very low precision matmul. And that's all that these things are going to do.
replyI looked at this for a client close to 20 years ago. The problem is that you need to get software support for it. Arguably easier these days with open source but still doesn't magically happen.
replyThe nice thing is that for GPUs you're rarely writing directly against the metal but instead pseudocode lowered by the driver; the driver could know to just lower it this way. On an NVidia GPU they control the whole stack and don't need to get a further ecosystem going there. Possibly the same for Apple.
replyApple is such a vertical stack, I'm not sure how to generalize from that. Certainly the same was true on IBM mainframes. And, when I wrote open source, I could probably have written abstraction more broadly.
reply