I don't see how anyone justifies a claude subscription. Do you want to rent the tools you use to making a living from the people who are trying to automate away your job?
https://village.dev/blog/closed-ai-models-now-dominate-the-p...
That's $104/day of opus 5.5 tokens, or $265/day of fable 5.1 tokens.
If you were able to saturate usage of this system, you could pay it off in 3 days of fable time or 6 days of opus 5.5.
But ok those are more intelligent. Let's go with something that scores similar - sonnet 4.6. 5.2M tokens is $78/day and the system is paid off in 8 days.
But you’re not generating Opus or Fable tokens.
And people buying subscription plans aren’t paying API rates.
I have a Claude subscription and I have hardware to run Qwen3.8-Flash-Next a full order of magnitude faster than the theoretical system you specced above.
In no way would I ever consider my local Qwen3.8-Flash-Next setup comparable to even a $20/month Claude subscription. It’s just not.
I do try to use it for bounded tasks from time to time, but using it for actual complicated work generates so many problems later that it’s not worth it.
Often I can have Opus produce a better result in 1/10th the tokens. The Qwen model often gets things wrong that require 10X as many tokens to debug or fix later. You cannot make comparisons like you’re trying to do.
In my opinion, what we actually have to compare against is the $20 subscription compared to what using Claude or ChatGPT is like with the free plan.
If I use Claude free it cuts me off from using it for hours after a very small number of questions being asked, and it can’t be hooked up to a coding system at all (no Claude code CLI or VSCode extension).
So when I dropped a $500 GPU into my gaming server in my closet, I’m breaking even compared to the $20/month Claude subscription in 2 years (and my GPU already appreciated in value since I bought it).
I can send as many queries as I want in any way I want, including custom applications that would have normally been billed at API token rates.
On top of that, all my data is kept private to my server.
(I’m not counting electricity because my server was already on serving all my self-hosted apps, so a significant amount of electricity is only being added to the load when I’m actively asking questions)
No, we’re not going to compare $500 GPUs to free tiers.
The contorted logic in these threads makes my head hurt.
> So when I dropped a $500 GPU into my gaming server in my closet, I’m breaking even compared to the $20/month Claude subscription in 2 years (and my GPU already appreciated in value since I bought it).
But the $500 gaming GPU in your closet isn’t providing Claude level intelligence or performance. Not trying to brag, but my LLM hardware is an order of magnitude more expensive than that and I hold no illusions that it’s comparable to what I get from Claude.
If you want to compare against something, compare against Qwen tokens purchased through a provider. Compare like for like.
The $20/month Claude sub is on a different level of performance entirely. There is no direct comparison on quality or speed until you’re spending A LOT more on hardware, and even then the closed models still win.
The free Claude subscription is an insufficient level of service because what I'm looking for is the ability to send as many queries as my heart desires, and to be able to use it with a coding agent. Therefore, I must spend some kind of money. The cheapest flat price plan I can buy from Claude is $20.
That's why I'm comparing against $20 a month. Of course, maybe for this sparse use case it would be better to use API billing instead of a $20 plan, although maybe that means I'm thinking about billing variability.
I am not comparing it to the level of intelligence that Claude offers. I'm comparing spending $500 one time to spending at least $20/month forever. Yes, it's smarter than the model I'm running, but if I'm not paying that money Claude ends my usage mid-conversation and tells me to come back in 5 hours.
The model I'm currently running is Qwen 3.8 Flash Next using Strata (https://github.com/Niko1221/Strata), the VM has 64GB of system RAM assigned (which was previously sitting unused, with deep apologies to Ram Altman). It's definitely not a bad model, and gets the job done for me.
I would generally agree with you that I'm never going to break even against buying tokens from better open weight models on the open market, or maybe even just buying Sonnet 5.5 tokens, but I'm also gaining the data sovereignty aspect, which is worth a non-zero amount of money.
On top of that, the GPU is an asset with resale value, so I'm not losing the money permanently. It has appreciated in value by $150 since I purchased it, and will probably be worth no less than $200-300 in the next 5 years in the worst case.
Doesn't matter if it's free, you'll spend 3x the time correcting its mistakes.
I think most people unfortunately don't care about any of those things, or at least they don't care enough as we are about to see by the trillion dollar IPOs of these AI megalabs.
Anthropic continues to serve haiku - which suggests people have a use for less-than sota models.
A bugatti is undoubtedly a higher performance car than a honda civic. Yet if you're commuting to work in rush hour traffic, both cars will get you there in the same amount of time - and one is going to cost you a lot less.
The pitch of the frontier labs is that you need a bugatti for everything. I don't buy that.
1 qwen output token != 1 opus output token.
Yes I know what the benchmarks say. It does not reflect my experience.
You’re also comparing benchmarks for the full model, but then telling people to run IQ3_S. I found the IQ3_S model to be a substantial step down from 4-6 bit quants.
> You’re also comparing benchmarks for the full model
I wish there were more concrete ways to judge the difference. I've spent a bunch of time using 3 and 4 bit versions of qwen 27b, and my conclusion was that the lower qaunt made more mistakes - but that both systems were able to recover from those mistakes in a coding context. I'm not sure if that holds across the board, but that's the best answer I have so far.
It's also unclear if anthropic always serves you f16 versions of their models. My friends who use claude a lot complain about intelligence that varies from day to day - but that's entirely anecdotal.
If all you do is generate tokens all day, sure, host your own infra. For people that aren’t maxing out hardware, using someone else’s is a pretty economical option. $20 with deepseek flash gets you a LOT of agentic use, and as the frontier moves forward with new models, so do I.
But let's compare apples to almost-apples! Qwen3.8 flash next is $0.47/M on openrouter, so $2.44 per day, so 246 days to break even. I say almost-apples because on this $600 system we're running an IQ3_S quantization. The hosted model is running... nobody knows what quant. It probably varies bases on the time of day.
Also, if I wanted to generate Sonnet 4.6-quality tokens, maybe your suggested setup is the better approach (though I still highly doubt it), but I don’t. I want Opus 5.5 / Fable 5.1 level of quality, because the ROI in terms of human-in-the-loop time saved and how much better and faster I can build is worth WAY more than $200 / month to me. That’s basically a rounding error for my business.
Anyway, if it’s worth more than $200/month definitely don’t spend $600 and expect it’ll replace that. But $2400 ($200/month for one year) gets you much closer to sota than $600 - and whatever the line that stops being a rounding error for your business gets you closer still.
How much is it worth to own your livelihood? To me outsourcing to anthropic or openai for $200/month means you’re handing them the ability to kill your business at any moment without even noticing. That’s worth more to me than 30% points of intelligence.
You clearly have an ideological issue with these companies, but I don’t share your views or your values. They could cut me off tomorrow but it wouldn’t kill my business. Worst case, I’d have to figure out how to make do with what you’re already choosing to do today.
That's just the issue of your access to AI in the worst case.
The actually impactful part of the worst case would be what happens to your ability to have a job, put food on the table, or even afford to live in a house.
One of my beliefs, call it ideology if you like, is that api cost is much more representative of real cost of this stuff. Good luck switch to local models the same week everyone else decides to switch because subscription prices went up 5x… it’s going to cost more than $600.
Anyway, like it or not, you are giving $200/month to fund another ideology. One that looks to me like a leopards-eating-faces type of ideology.
As long as you have a business. AI will "Sherlock" many and commodify all.
I don't really understand the argument being made here. The best models are better, but they're not that much better. If I'm paying only 20% as much for 20% less performance, I can make up the 20% and have abundant cash or compute left over to spare.
The other thing that matters to me is the tooling. I could use Deepseek Harness with Claude but I'd rather support the same people that brought me DSH.
As a consumer I don’t care. The price I pay is how much it costs.
I’ve been hearing arguments that they could increase the price at any time. For years.
Instead the opposite has happened: A $20/month subscription has become more capable than it was 6 months or a year ago, by a huge margin. I can get so much done within a day of the $20/month subscription that it’s hard to believe it’s so cheap.
By the time they reach this point of raising prices, if they ever do, the cost of models with current intelligence levels is going to fall a lot.
So all of these arguments about buying hardware now and switching to open models now to prepare for future price increases are still wrong if they’re argued on the basis of being locked out of cloud intelligence costs.
The cost for a given unit of intelligence is only going to decrease.
Seems more like a final fight against open weight models than a sustainable business model.
Seeing comments about the major AI labs breathing their dying breathes because they have so much debt is really fascinating. The ideologies have started to exist in their own little bubble divorced from reality.
Certainly I see the benefits in terms of privacy and security. But if you're trying to justify local models on economic grounds, you also need to factor in electricity costs.
(Also, people will say that "less than a year ago" represents an enormous capability gap. But people were already ranting and raving about how capable the models are and how cooked the industry is, considerably longer than "less than a year ago"; so.)
But ok, let's compare: https://benchlm.ai/compare/gpt-6-astra-vs-qwen3-8-flash-next
Astra's score: 85.8/100
Qwen3.8-flash-next's score: 55.87/100
So there's a ~30 pt difference between them. Let's add 5% more for quantization damage.
My perspective is, yes - that's a huge difference. No - it doesn't matter. Flash-next is above the threshold of useful intelligence. I can ask it to do something and it does it. Sure, it burns some tokens on bad ideas on the way, and sure it can't turn a single paragraph prompt into million dollar business. I don't need or want that - I want something I can ask to refactor a module, and it does it and it doesn't stop until the module compiles and all the tests pass. flash-next does that.
Don't have to be. They just have to be good enough for the tasks you need.
I use both models extensively and there is no way I would ever agree that Astra and Qwen3.8-FN are that close in the real world.
I agree with the other comments: These comparisons are silly.
You keep saying that you don’t understand why anyone would have a Claude subscription and reciting ideological reasons not to. I’m assuming this means you don’t use their new models much? That’s the only explanation I can think of for how someone could think that Opus 5.5 or even Sonnet 5.5 and Qwen3.8-FN are close.
I use them both (often Qwen3.8-FN as a subagent) and any claims that they’re actually pretty close read as downright delusional to me.
Putting aside the absurd straw man, you’re touching on an important point: these small open weight models are sufficient for how you like to work, but it’s not remotely good enough for how I like to work. Opus 5.5 is barely sufficient for how I like to work (many parallel work streams, as little of me in the loop as possible, etc).
So not really work, then?
You’re automating yourself out of a job using capital you don’t own, and you think your position will hold? I am skeptical.
I work like the parent poster does. I am working harder than before the AI tools came around. I am just working at a higher level. Not too dissimilar from someone moving from programming in assembly language to significantly higher level language.
> as little of me in the loop as possible
You may be confused about this. If I am running just one loop at a time, you'd be right. What modern tools allow me to do is think more like a supervisor that has dozens of engineering teams available to him. I always have a long list of problems I need to solve. I used to sit with one and bang on it until it would pass acceptance criteria. Sometimes that took months. Now I just farm out each task to an agent orchestrator, provide it details on what I want done, a bit of design and architecture and acceptance criteria. I then step away from it and move to the next agent orchestrator and repeat the process for the next task. I usually have 6 or 7 large tasks being worked on at the same time. This is currently my personal limit. I can't babysit more than that. But as the models and harnesses get better I suspect I will have to do less babysitting and this process will get more autonomous.
It may sound like this process would remove the need for engineers. In my organization it is doing quite the opposite. As the organization figures out that we can do a lot more, we change our plans to build more. A lot more. Building still requires engineers, more than before, for us. But only those who can think like architects and can manage large teams of AI agents and orchestrators that work 24x7.
And I work for myself, so no, not automating myself out of a job.
Although I suppose I’m automating the people I won’t hire now out of a future potential job.
I run my own local models, and I can tell that yes, one ask and they do... with many more errors, and much much slower, compared to the lowest levels of frontier models.
Sure, if one has no standards at all, local LLMs definitely are "above the threshold of useful intelligence".
I'd rather wait all day for an impeccable implementation of a single feature than get 10 features worth of "okayish" code
I have hardware to run Qwen3.8 Flash Next (the model you used in your example) much faster and closer to full quality.
I do not understand how anyone thinks it replaces a $20 Claude subscription in performance or quality. At best I some times have Opus give well defined tasks to Qwen3.8, but I have to be very selective about what is handed over. I also use a lot of Opus tokens reviewing and fixing it.
For real work (not fun projects or experiments) I won’t even bring the Qwen models in to play at all. The gulf between Opus and Qwen3.8-Flash-Next is that big.
So I hope everyone takes comments like this with a huge grain of salt and healthy dose of skepticism. If your work has a high tolerance for slop and you can wait a very long time, the local LLMs can do a lot.
However claims about replacing a Claude subscription with Qwen3.8-Flash-Next? The only way this makes sense is if someone is choosing on ideologies like privacy or avoiding big corporations first, with quality a distant second.
I get downvoted every time I say this, but it’s objectively true. The open models that can be run on consumer hardware are far behind Opus, and quantizing them to Q3 takes them even further behind than the benchmarks show.
Also 100 tok/s prompt processing and 20 tok/sec is painfully slow for real work. The slowness is multiplied by how much extra work the less intelligent models have to do to get tasks done.
I want more than "functional"
In fact, I just switched from a 96 GB ram laptop to a 32 GB ram laptop specifically so I can have GPU acceleration because 20 toks can’t replace the 6000toks or whatever astra does
Not today. But when it turns into a $600 brick you plug in next to your router, that's gonna work for most people.
For today, I'm speaking to software developers - who can 100% handle setting up a local AI system - it's only as hard as building a PC.
I asked an AI agent to setup local AI inference for me. I gave it my use cases. It downloaded all the required bits, installed and configured them. Benchmarked different settings and models for my hardware and my use cases and then produced a nice report with findings and recommendations.
It is significantly easier than building a PC.
I know that configuration works, because I am running that configuration right now.
The whole AI world is run by sociopaths that want to make human brains economically uncompetitive comparedto computers. If you think running local models will save you, you're either very pessimistic about the future of AI, or you haven't thought it through.
Also, I've been playing with speech models for the past 2 years, and for me anything below 8-bit is a huge drop in quality; so quantisation for LLMs and for other bespoke modality models, is not 1:1.
[1] "Reliability Scaling Laws for Quantized Large Language Models", TUM, https://arxiv.org/pdf/2607.10855
It explicitly says that Quantization-Aware Training is out of scope, may reveal different trends and has potential for extreme quantization. Heavily-quantized model releases today generally use QAT or something similar, so I don't think the paper can be universally applied to modern quants (e.g. [0]).
That said, I don't see any evidence that TFA does any of that so the paper's claims may be quite applicable there.
[0]: https://huggingface.co/vcruz305/DSV4.1-Flash-1017-SAGE-REXL3...
If that scales with parameters I don't see why QAT shouldn't be the norm.
(As I was only comparing the QAT f32 model vs the quantization its possible that a not QAT f32 would have performed even better)
I expected it to be the norm since GPT-OSS
that's why different models exist.
If you think of it from an informational perspective, you can imagine that text tokens are much more high-dimensional than a transformed/down-projected variant.
With this reasoning the central layers are mostly building onto the residual stream, and can do fine with 1b and 2b weights.
I have never seen it neither on the link above nor on HF.
Ollama will expose the metadata on models too, showing the quantization for each individual tensor.
Other techniques likely beyond the scope of that one utility allow you to quantize some weights more and some less. So most model teams put our "average bpw" numbers.
On HF, click on any Q2K quantized model in GGUF format, it will show the metadata. Look at the expansion of the blocks. You will a mix of quantizations.
For comparison, the whole thing takes 1.8 sec on my RTX 6000 rig with GLM5.3-Flash, and 12.5 seconds on my RTX 5090 rig with Qwen 3.6 27b (it overthinks too, unlike GLM).
I can't imagine waiting 2 minutes for each small turn. It's like... 1 coding task per night? Sounds like, as an engineer, I can accomplish more in that time if I did everything myself manually without any AI.
The main/best setup I have running so far is qwen flash next with halogen-server.
But if anyone knows if and how one can run even more impressive stuff, please let me know!
I haven't done work lately on Dwarf Star. I got it running a few weeks ago and had my agent make a couple of modifications to it to get it running, but it wasn't materially better than my lemonade setup. I might revisit after this post, though!
The happy spot for these is for long running closed loop tasks where I’m stepping away for the evening. It doesn’t matter as much if it finishes in 1 hour or 10 hours as long as it’s done by morning.
For tasks where I’m sitting at my desk, it’s not yet where I’d consider it comfortable to use.
I do think this groundwork is going to be awesome come M7, M8 and so on.
I've had decent results with gemma4:26b (although it's more like batch processing when used as a coding agent) and my experience so far is globally in line with the first two sections of the ARCHITECTURE.md: fast with small contexts but performance degrades fast as its size increases. If the rest of the contents of that document are true, I'll need to retune my setup from scratch.
You mention "See later in the reply for the prefill at ~300 t/s" but I didn't see that in the video nor in any of the Twitter replies, could you link where that is supposed to be happening? Is that warm context perchance?
So I implenented a task queue so when I do have all the ideas, I can just set them to work, and "chew" through the work.
I get that you can concievably get 100 agents to do something interesting, but 99% of my work doesn't seem to need to divine naiver stokes.
Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is.
Lol yeah of course, that's not the Apple hardware showcased in the video. Sure, my 6000 Pro also does 11K in prompt processing, but my comment wasn't generally about local models, I thought that was obvious.
> I get that you can concievably get 100 agents to do something interesting
Uuh, OK, cool. Let those vibesloppers do that process if they'd like to, I think most professional software developers already found that process to produce slop and not production-worthy software, so we continue with our process that produce code of quality instead of aiming to produce as much as possible.
> Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is.
Huh? Where are you getting this from? Where did I say anything of the sorts?
Comparing it on Nemotron, it runs just about the same as plain old llama.cpp on the cpu (except prefill seems much faster). I don't see an option to set up DeepSeek in "modchip setup", so I haven't tested that one yet.
Edit: this is on a Ryzen 7945hx 16-core Zen 4 + 96 gb DDR5.
> llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.ffn_gate_tid2eid.weight' has wrong shape; expected 4, 129280, got 6, 129280, 1, 1
And running the bundled llama-cpp instance from the download, as shown in the readme, fails with "Illegal instruction (core dumped)". So I have no idea how to run this.
Where that threshold is crossed is up for debate.
Qwen 3.8 27B Q3 readily beats Qwen 3.6 27B FP16. Difference in hardware requirements and token rate is of course massive.
EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.
haven't tried it yet though.
But yeah, 32-64 is the gateway I think. If Qwen4 can make a leap in the flast-next MoE and hit 64 GB on VRAM w/context, that's definitely the cloud killer to me.
You don't need papers, good quants are usually published with benchmark data, for example: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-... which maintains parity with FP8 even at 2 bit.
The ISTA accuracy looks good on that page, but it's only a chosen handful. Instead, look at most of the graphs on Unsloth's Qwen 3.8 Flash Next - https://unsloth.ai/docs/models/qwen3.8-next
As always, check the quant vs KLD graphs (half way down that page).. I found that not all quants are the same - Unsloth's GLM Q4 -> Q3 degrades a lot faster than their Minimax M3.1 Q4 -> Q3 ggufs.
(and before you crap on it, this is not the same as the original Bonsai, this newer one is much better, and upstream support in llama.cpp etc. is being worked on)
> Shrinks DeepSeek-V4-Flash from 76.87 GiB (which crashes in OOM on 64 GB Linux) down to 56.56 GiB
And later
> Standard IQ2_XXS (76.87 GiB, Unpatched)
Except unsloth’s IQ2_XXS GGUF is 91GB, even their smallest IQ1_S is 82.5GB. So I wonder which model/quant the number even refers to.
Also the mentions of “compression” and “MoE pruning” also make it sound lobotomized even more than unsloth’s IQ1 quants.
Also, just noticed that “moe pruning” likely means “the model is supposed to run 6(DS)/8(Qwen) experts per token per layer, but we only run 4 to save on time”.
Either way, the final gguf is 75% of the size of unsloth IQ1_S, which by itself should tell you a lot about its quality; they (somehow; I have no interest digging that info from that slop paper) shrink the (already IQ1) FFN weights, and shrink some attention weights from Q5 to Q2. I’m surprised it was even able to generate the text in the demo.
Reddit is flooded with these breathless headlines “xyz tokens/s on my light switch”. It’s always quantizations below 4 bits per weight. Mostly useless.
I wish people would just write their readmes themselves.
- Bonsai Qwen 3.8 27B
- Strata
- Pulsar
- This...
I just wish upstream Llama.cpp would accept the PRs faster - they build multiple times per day, but their PR backlog was 1.7K last I checked, which is a ridiculously impractical number to review with a few volunteers.
I watched the Qwen 3.8 Bonsai feature and it took over 3 weeks from creation to be reviewed, and even then it doesn't include the patches for bugs, that they fixed downstream and simply gave up pushing to upstream because the process was so laborious.
So they end up with delayed or missing model support, and half working implementations like Bonsai's Hadamard Folding, and new models lose out on this kind of optimisation.
I suppose this is fine if you intend your project to be the reference project for forks and encourage this kind of specialisation, but I don't think that's intentional.
At least I can use llamaswap to juggle the inference engines and models on the fly, and abstract the fact I need to install 3 or 4 different ones.
How would that get fixed? I'm skeptical of lowering the standards of review. I suppose if you have some source of trusted volunteers...?
When if comes to the human reviewer unless an issue is structural, don't care so much about the formatting and keep to one human approval. For simple changes, like documentation, comments etc, these can be approved by bots themselves as any fault is non functional.
If you consider that if bugs are noticed they will be fixed by other PRs, it makes less sense now to try to fix all issues in each PR, especially as there's no guarantee this would cover all of them anyway with so many ongoing changes.
At some point they will have to cut their losses, strip back to a week's worth of PRs and close everything else to get back on top (perhaps this is actually what is happening, it just takes the bots a while to chug through them).
Edit: just checked and it's now 1.9K PRs.
It is about $10 per person per month for enough processing power for useful work with data sovereignty.
Steady 12 to 14 TPS.
Not blazing, by any means, but fast enough to get meaningful personal work done, with a power consumption of 15w.
I remain exceedingly skeptical of people who are claiming to run fairly large models in ridiculously tiny sizes. For comparison a 'regular' version of v4 flash 0731 has file sizes more like this: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
Even the 97GB Q2XS will show very significant divergence from the full precision thing. Running a lobotomized model just to say you're doing it is really not accomplishing anything special.
I can imagine Open AI and Anthropic( among others) artificially manipulating the pc part market to keep consumers from buying capable hardware.
As is a ROG tablet is available for 3K with 128GB of ram. I don’t need to beat Opus, I just need good enough
The readme is practically unreadable ai slop as well
frontier labs are going to need to have some new tricks up their sleeve that are inaccessible by prosumer level compute