> A Mac Mini can run one LLM inference thread at a time.

That’s not accurate. With MLX, at least, parallel inference is both possible and useful. Model serving tools like LM Studio and oMLX support parallel generation with continuous batching, and the total throughput increases with it.

I think local inference will be fast enough

There is so much happening in that scene, where tokens/sec double or 10x

So I could see the same hardware doing 20 tokens/sec on a large model suddenly doing 200 tokens/sec in the future, a better device in the future doing 500 tokens/sec, while having vision models baked in, audio models etc

Users wont consciously switch to local, they will just have it and use it

You can always buy multiple Macs. I think Apple's great differentiator could be making frontier class models local. I you want more threads, buy more Macs.

I don't want to run any workflows on someone else's computers.