As someone not in the loop, can you explain more of what you mean.
replyday one support for all inference engines so that whichever kind of capable computer you have, you can run it.
you do need a really capable computer though
reply> you do need a really capable computer though
replyThey suggest a pair of Mac Studios w/ M5 Ultra chips and 512 GB of RAM. A single Mac Studio w/ M5 Ultra and half that RAM (256 GB) is around $11k, and the 512 GB option isn't even available yet. So.... yeah.
You can just use your old dual 512GB M3 Ultras until you get the M5s. Man, people always whine about something...
replyThis comment has the unusual property of being applicable whether or not it's sarcasm!
replyO shit, I didn't even think of that. I've got a few of those kicking around, honestly I was just using them as doorstops at this point.
reply[flagged]
replyAnyone who has tried a smaller/quantized version locally on apple silicon? Would love to know your experience, also tk/s on what hardware specs?
replyYes, but still, what about users looking for NPU support. What happened to Microsoft's copilot+ PC campaign? (the NPU hardware, not the logo key and ad campaign)
AMD's XDNA2 NPU is best supported by ONNX, I think Qualcomm Hexagon is too. The best consumer unified architectures are Apple, AMD ai+ and Qualcomm Snapdragon X Elite, even intel has an NPU offering.
replyNPUs are meant for tiny, tiny models that can run basically at all times but cannot affect battery life (e.g. translation, live-video cleanup, filesearch etc.), not for big LLMs. This is about 500 times to large for a NPU.
replyNPUs you can find in AMD high-end CPUs are closer to specialized DSPs (basically big VLIW/SIMD vector units).
replyPeople love doing heavy math stuff with them (https://destevez.net/2026/05/getting-peak-tops-on-a-ryzen-ai... - not my blog)
There are plenty of smaller less powerful open weights models if all you have to run it on is an NPU.
replyThis is a much larger more powerful model, so running it locally will not be for everyone, but you can still run it via a hosting service such as Fireworks AI.
I was mistaken in my thinking that open models are able to be quantized to run on average consumer hardware, I think they have outgrown that concept.
replyI guess that's the attraction - that they may be small enough (likely quantized) so that you can run them locally, so there is no privacy issues, and no-one can turn it off, etc, but technically "open weights" is just referring to models, of any size, where they give you the weights (basically give you the model) so you can run it yourself, instead of having to access it via paid API running on some computer that they control.
replyCopilot+ PC is such a terrible and wasteful marketing effort that Microsoft has given it up.
replyNPUs are not inherently useless, but definitely so the way they are forced onto users machines.