Doesn't this depend on the model and quant as well vs. the hardware and its capability? Plus memory bandwidth, amount, and layout.
replyMy Strix Halo now can run some models through the NPU and its roughly as fast as its GPU+CPU (on some of the models - some its slower)
Can't you just look at memory bandwidth for generation?
replyYou also need to look at memory size (to store the model). And processing can still be the bottleneck.
replyIt’s more nuanced than that, but yes that’s the biggest factor.
reply