1. Most useful LLM work is done in parallel. A Mac Mini can run one LLM inference thread at a time. The cloud can spool up dozens and spread that inference across efficiently batched operations over a fleet of hardware.
2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally. But the advantages of running >5x the token throughput per thread can’t be overstated. Add in the multi-threading advantage and it’s a knock-out punch for local LLMs.
Local inference has a role: if you’re working with extremely private matters or you want an uncapped model that will talk dirty or generate NSFW photos, local is the only option. I think Apple and others will continue to also run a lot of useful workloads locally such as text editing suggestions, speech to text, text to speech, and image manipulation. As local hardware improves, these capabilities will get better too.
But, for most LLM work, the cloud will continue to dominate for a long time to come, if not forever.
I'm fatigued by it all at this point. It's streamlining the interesting and fun parts out of my job (by practical necessity of use there), and if I used it half as much outside of work I'm sure it'd do the same there too.
This is the prevailing opinion of people even outside of tech.
That said, I think it's a good thing that this sentiment is coming to the forefront.
He was complaining that he would ask how to perform a certain repair on a car, and the LLMs he tried (ChatGPT & Grok) would give him a long involved process and he'd ask why not do it this simpler way and it would say, oh you're right! He just found it gave bad advice and realized (rightly) that in areas he has less expertise in he has no way to judge how good the outputs are.
This is from a guy who loves tech, historically worshipped Elon, loves his Tesla, and (rightfully again) didn't buy into SpaceX because he thought it was overvalued.
In the past when I visited for holidays he was liable to have a positive outlook on LLMs and their utility. Seems telling that he's starting to see the cracks.
This is easily the biggest problem with the current models. The models are just way too eager to please / say yes to the point that the models are happy to lie/make shit up if it means it can say yes.
We are both late and early.
Buy an Nvidia Spark, then whatever cheap Mac you want to use as a thin client. There's no reason to force Apple Silicon's round peg into a square hole like AI inference.
Apple is doing something very different. Their AI experience for end users definitely has been a little behind.
Apple Silicon, however, has been quite unique for the last 4-6 years and it's increasing overlap with LLMS.
The model/chip optimizations are definitely improvements, the thing that is really standing out the past 2 years is how much the open source model community has been making possible, especially when you know a group of use cases.
You think this is a mistake...
Of course. Do you think this was on purpose? All part of Apple's brilliant master plan?
The one thing that is marginally exciting: the Apple SoC or M series chips.
It's unfortunate they are locked behind crappy macOS and other proprietary apple crap.
Unsurprising. Apple seriously thought the iPad would replace computers and usher in a "post-PC" word during their "What is a computer?" ad campaign era. Now they are sticking phone chips in laptop chassis.