And now if you stretch you can run DeepSeek Flash 4.1.
And if you were willing to roll up your sleeves you could build your own AI lab at home and run this model and models like GLM Flash 5.3 and DeepSeek Flash V4.1 at lower quants. The budget GPUs (like V100) are kind of not budget now, but still <5K to acquire 128GB of fast VRAM that can compete with a 5090 and actually has a lot more interesting options available IMO.
Dense models also don't like quantization as much and are less tolerant of it. There are some very interesting smaller models like https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4... Dory that I would check out. You need system RAM or to stream a larger MoE model from disk and it won't be fast. You need to specialize the models some to get quality results if you have very limited system resources.
I asked Claude to come up with 7 problems to solve that were not part of any benchmarks. Besides speed, I asked it to put together a scoring that would reflect models that perform well on the 7 problems and in a reasonable time-frame (RWS, the definition is in the artifact). I valued models that produced good answers in a reasonable time better than models that had very high throughput but kept spitting useless stuff. Overall, IIRC most of them still had a >30% chance of producing wrong/non-working replies.
I then tested Strata ( https://github.com/niko1221/strata ) and the results are not even close. You can basically ignore everything in that table and use Strata instead, you can use bigger/less-quantized models and have way better results. Basically any otherwise-decent model degrades noticeably at Q < 5, and KV cache at Q4 also contributes a lot.
Still, a 20$/mo Claude subscription is way cheaper/more efficient if you code 2-3h in the evening than running stuff locally, unless you really have hardware than can hold models that get most things right at the first try. You can iterate way faster and cover a lot more ground in 2-3h using a Claude subscription, or OpenRouter if you prefer to change models. You get faster replies that are 99% correct at the first try, whereas using the local model you'd be spending 30% of your time fixing things that don't work and prompting it to do better.
Nowadays I use the local model to delegate tasks that would burn a lot of tokens, like sifting through big data archives for indexing and such.
This is not about if a frontier AI provider is cheaper. It is about having control over your data and use of AI.
https://stealthgoat.com/fnfl/pelicans.html
Also YMMV in “faster” — I have less than 5K into my home AI lab shenanigans and it’s pretty fast :)
Yes, but like original poster I have 16GB of VRAM and 64GB of RAM, I was referring to non-Strata benchmarks :)
I did quickly run a couple things on Strata but haven't gotten around yet to benchmark bigger models/customize them with Strata, but will do rather soon. So far Strata/Swift IQ2_XS performed really well compared to non-Strata/Qwen3.8 quantized variants.
> This is not about if a frontier AI provider is cheaper. It is about having control over your data and use of AI.
Yep, totally get it, and to be honest getting to run ablated models sometimes can also be useful.
> Also YMMV in “faster” — I have less than 5K into my home AI lab shenanigans and it’s pretty fast :)
My rig is... ~1.5k, maybe 3k if we count the CPU and GPU upgrade since 2019 :-D I really wanted to grab one of the latest mini-PCs (Halo, Spark, and other minor assemblers) for local setup, but to be honest I can't justify a 6k+ purchase for me and my wife alone. Perhaps in 3 years these mini-PCs will get to the ~2k ballpark (perhaps used), and although older by then, I might be able to run things 24/7 locally as well.
Get a couple or V100S GPU. Figure out cooling. They will run it /fast/ with the right engine. :)
It will show you on the model selection IF a model fits and if you use a "Qwen 3.8-35B-A3B" type of model, it should just work out of the box. A3B stands for Active which means you need a lot less fast ram.
You will run into issues fast though: If you have big tasks for it to do, your pc/mac will get warm very fast and noisy and you will cosume relevant more energy.
And as you mention: yeah small context!
Its still not that practical in my opinion.
https://github.com/Niko1221/Strata/
My setup is a Ryzen 3600, 90GB DDR4 allocated to a VM (don’t even really need that much, just had it available, special thanks to the year 2024), and an RX 9060XT 16GB ($500 GPU).
I get around 40-60 tok/s depending on the quantization level I choose.
As you can see, the CPU is nothing special, don’t even need DDR5.
So it depends on what you're doing. Using it for a fun oracle without feeding it too much context per turn? Sure, an older Max chip is great.
As a coding agent doing meaningful work on a sizable project? Compared to a cheap as chips cloud option, it will take an order of magnitude more time to complete.
Isn't it also rumored that the M7 is reworked for AI?
And Xiaomi announced their AI Cube with 1.2t of bandwidth. Just because Nvidia is protecting its business by giving consumers either compute or bandwidth and Apple being something in between, doesn't mean we can't get both a lot better balanced.
I'm gonna max out most of the specs on the next one I get, would feel insanely wasteful to do that now.
Then again the people working on stuff involving this much local model compute demands probably make enough where they don't care about the couple grand they'd save building their own node. I guess apple will get the bag from being the nicest prebuilt out there in this case.
This is the first time I’ve ever kept a computer this long and never really felt like it needed to be upgraded. I suppose this is a bit of a gift and a curse for Apple.
I don't know man?
The generation kind of does matter. I'm not gonna lie, matrix multiplication units speed things up considerably. That hardware confers a material change in capability.
Maybe from M5 up, it doesn't matter? But you're definitely going to notice the difference between M1 and M5 for instance.
It feels just as fast as day 1 and I've never had any speed issues. Say what you want about Apple and their pricing and ecosystem but I've been using their hardware for over 20 years now and it's always been top notch.
If you buy their support/AppleCare, your support and warranty is excellent too. They don't skimp. I've seen them go so far past the extra mile it's unreal.
Definitely grabbing 128 GB + for the next machine.
I jest...but I'm disgusted by the price gouging going on. Glad my hardware is relatively recent and I had mostly maxed out already when I bought it a few years back. The same setup today would be closer to $7k with equivalent parts.
I am one of those: bought the M1 Max with 64gb thinking in running VMs, afterward bought a lighter Air with less memory and ended up using the M1 Max as a kind of luxury for LLMs and multicore computing.
The only hope we have is the AI bubble bursting, but that will undoubtedly come with its own set of macroeconomic surprises.
The fabs are terrified to overscale in case the market collapses before the builds are complete.
> AI datacenter provisioning will go down
Depends on political opposition at both local and national levels. Hard to predict the future, which feeds into the previous point.
Supply is not elastic and responds very much slower than demand. There's no conspiracy or corruption needed for this result.
Have you not seen Micron or Sandisk stock?
Micron's stock price did go way up, after they stopped selling DRAM to the consumer market. It has been flat for several months though.
For comparison, SK Hynix stock is flat YoY and Samsung's stock did go up this year, but has come back down a bit (they're also much more diversified). Those are the only other companies outside of China that actually make DRAM at scale. There's also CXMT, which is making inroads, but its stock is not traded on any open markets, and the company is heavily backed by the Chinese government.
Mini PC with Occulink, EGPU dock, 750w power supply and a refurbished 3090 is 2500. It can run Gemma4:31b at 1000+ tok/sec prefill, ~40 tok/sec generation (which is slow, but a fully connected model is way better for all tasks)
if you want more specialized partial activation models like Qwen, you can get 100+ tok/sec generation.
This beats anything that Apple has by a long shot. You don't need to run large models for usefulness. Large models can get things write the first time, but thats their only advantage. Running small models that are good enough like gemma4:31b fast allows you to do faster iterations.
And if you really want, you can also do multi GPU setups if you want slightly slower model loading with usb4/thunderbolt ports.
TLDR; local ai has already been useful for quite some time if you arent brainwashed into believing that Macs are the best for running local inference.
Before you make assumptions, there are valid reasons for the choice I made:
I paid $1k for a refurbished M1 Max 64GB macbook that can run Qwen 3.8 27B at 24tok/sec (looks like they are about $1.5k on ebay). In my case, Apple was a lot cheaper (used) if you're looking at price-per-token. Yes, it's almost twice as slow as a 3090, but it's also half the price /performance ratio.
Qwen 3.8 27B Q4 with 64k:
3090: ~40tok/sec
M1 Max : ~24tok/sec (MLX)
In fact, the Max can even do 265k context which isn't possible with a single 3090 last I checked (without dropping to Q3-Q2)
Try more like 70-130 tok/sec. https://github.com/L-Forster/open-jet/tree/master/megakernel.
And also Qwen models famously chase benchmarks. They are good for writing code, but fall behind in other areas. Gemma4:31B is 6gb larger and does a much better job across all areas. https://www.reddit.com/r/LocalLLaMA/comments/1vyzopv/gemma4_...
Go ahead and run Gemma4 on your M1, and you will find that its unusable at 12 tok/sec.
And this is also not touching prefill, which is the biggest thing that you need fast, and Apple sucks in this regard compared to nvidia cards. With gfx cards, you can throw big codebases at the local models and it will happily crunch through them in very little time.
Sure, spending 1.5k on a Macbook enables you to run a local model, and if you really have a good reason for running local (like running heretic/abliterated models) for tasks you don't mind waiting for, it works. The difference is, spend 1k more and let you never have to pay for tokens, and you can run inference all day long without your computer thermal throttling, like your Macbook.
And I don't need to make assumptions when you say idiotic stuff like "We're getting close to local AI being really useful", like you have any idea what you are talking about. On par with being an Apple fan though.
I'm one of those Mac users, but I do have a linux tower for remote dev work and LLMs.