It is debatable if we will actually need that many parameters though, since recursive nets like HRM (https://huggingface.co/sapientinc/HRM-Text-1B) don't need to parametrize as heavily.
But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.
https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text
You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done.
Yes, with top-tier GPU farms you can hit hundreds of tokens per second. But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
Keep your mental context in your brain is critical
If I spend 10 minutes reading an article, that would only generate 3000 tokens.
That’s not counting the prompt processing time.
We have very different expectations for LLMs if your tasks only take a couple thousand tokens and you’re happy waiting 10 minutes for it.
> Yes, with top-tier GPU farms you can hit hundreds of tokens per second
My 5090 gets hundreds of tokens per second with this model. No farm needed. I’d have to double check but I think even a $1000 Intel B70 might break 100 tokens per second.
> But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
If that old Xeon pulls 200W from the wall and you pay national average electricity costs, it’s going to cost $0.90 per day to run it.
I would rather pay a dollar per day, get my answers 100X faster, and not have an old Xeon heating up my house.
This is an option if you must run local inference, you’re not sensitive to speed, and the budget is low.
It’s not going to be cheaper than paying API prices for the model though.
Let LLMs write the corpo code, as it will be unlikely to still be running in 5-10 years. Frontier AI is already at the point where it writes fewer bugs per LOC than humans. By a lot.
Go ahead and do your bespoke coding on your side-project loves and core libraries... The stuff that will last, anyway.
But if you're working for a corpo and still doing bespoke... That's... not gonna last, I'm afraid. Well, either you remaining there, or that, as it were.
At 5 tokens per second and unknown prompt processing speed, you may need a very extra long lunch break depending on your codebase.
These are fundamentally different tools for entirely different applications. They only look similar to people who don't understand the tools or their purpose.
It is common for agents to just stop because overload or some API error hijinks.
Or you get a TUI question that is blocking.
In general you’re right though, staring at tokens from agentic is not time well spent.
Some of these I’ve built custom harness around in iterm2 though.
I watch tokens to see if it goes in right direction. If model goes off the rails, then it is time to stop and adjust prompt.
Instead of coding agents like tool for local models I would like to see more "docker ps" like tools where you can queue up tasks that get processes incrementally (maybe when the pc i idle) and are specialized for doing retries and caching as much work as possible to work decently with slow local models.
It says so right in the readme. They’re not hiding anything.
Pending any new hardware or radically different LLM architectures, we're going to be waiting a lot longer than 2027 for 200B models on local hardware. SOC platforms like Apple Silicon are hamstrung by their obsession over raster performance, the hardware lacks the fundamental GPGPU hardware to be a replacement for real-world inference.
I am curious about the decision to not use GPU since this is for Apple Silicon.
Wouldn't the GPU potentially accelerate the DeltaNet/attention layers and matrix multiplication in general?
This prediction alone isn’t useful at all without a bound on speed and maybe quantization.
You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.
If you’re saying that hardware will catch up by mid-2027, I disagree. The limitation is fast memory and that’s going to be expensive for a while. I have a 128GB unified memory machine that can technically run 200B MoE models with enough quantization, but it’s so slow that there’s no reason to do it. It’s going to be a few years before we have enough RAM and processing power in basic consumer hardware without spending as much as a used car to get it.
> This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
Sorry, but 9 tokens per second with a slow prompt processing speed is not decent for anything other than getting short chat responses.
You’re also not running the full GPT4 quality model. I’m very familiar with that model from some other work and the 4-bit quants are just not as good as all of those KL divergence plots would have you believe.
You also have very short context. It’s basically useless for anything more than short chats where you’re okay watching the output come back at reading speed, skipping the reasoning part (which is important for calling it GPT4 level quality), and waiting a long time for the first token.
Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.
What are the LLM standards?
Do you know how many people use perplexity? I know many people who are not software engineers or tech workers and have a LLM subscription for rewriting their stuff (non-native english speakers) in english. There are many use cases for running good models locally. Maybe not for you, but someone might find this beneficial.
It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that, seems there was another collaborator ;)
Show something you've built with the model+tooling instead, truly dogfood it. I'm sure you'll discover things along the way too!
that would have run into a race condition unfortunately ;)
but there is a sample landing page + a python function on the repo which shows what the model produced. my goal is to integrate the local model in my workflow so that claude/OAI can call this model for basic stuff.
Not really, you start small, bootstrap as soon as you can, and off you go. Requires a good model though ;)
That is no where near decent at all.