> I’m not sure how much effect you get from higher TOPS with such low bandwidth though.
replyRaised arithmetic intensity. It would be quite good for LLM prefill, and decode would want to adapt by pushing parallelism higher (saving on memory bandwidth by reusing the same layer weights for multiple streams) or, if unable to do so (e.g. due to KV cache constraints) by resorting to speculative decoding (where the expensive 'validate' step handles tokens in parallel within a single stream). It does not make the platform inherently unviable. It would also do well at other tasks such as image/video generation, which is already quite compute bound.
deleted
replyApple M6 has 80-100 TOPS.
replyQualcomm Cloud AI 100 Ultra has 800 TOPS, almost 10 times more.
This could be very fast at running embedding models (if it supports the right modes)
replyConsidering that memory bandwidth is the most important for inference for most neural networks, that matters more.
replydeleted
replyModern inference stacks use different GPUs for decode versus prefill. This is a GPU suitable for prefill.
reply