Oh, interesting. So beats on prefill and matches at decode. MTP will help you a bit once you have it. I've got a lot of work to do to optimize prefill.
replyKinda wish I had a Strix Halo here to play with as well.
I just took a minute to look at your eider repo, very cool. It looks like most of the code outside the kernels and immediately surrounding plumbing would work well on AMD APUs, and probably also on Apple and newer Intel.
replyThanks; I have another, currently private, repo that targets both pure CPU inference and wgpu (to vulkan.) It's somewhat similar but... different. Shares some common pieces but needs to be refactored to share more.
replyBut it's been hard to make it competitive with CUDA. At least on this Spark and my only non-NVIDIA machine (which only has 16GB unified relatively slow RAM.)
I know a little bit about this problem space from previous work (we were working on performance-portable deep learning back around 2016). The infrastructure has improved but as far as I can tell not many teams have really "squeezed the toothpaste tube" and worked through performance issues systematically. These days a small team and robots can probably do it though.
replyAt my day job I may get access to big AMD AI iron in a couple months (to do research/performance tuning with). That could be interesting. Though that's likely to be of a very different shape from consumer Vulkan. I'd still like to have a Strix Halo to futz with. But I'll wait for RAM prices to drop. (Hah!). I do have an older BC250 board lying around but that only has 16GB RAM.
replyI got prefill up to 190 tok/sec just now, BTW.