That said, even the foundational models fail at the hard parts of my code so I use it opportunistically.
I have reduced down to just using zed, will three locally hosted models.
Qwen 3.6 27b on 1x3090 llama.cpp with 128k context ~50tps
Qwen 3.6 35B-A3B on 1x titan v + 2x1080ti llama.cpp with full context ~30tps
GPT-OSS 120b on pure cpu (slow)
I just use zeds parallel agents, task switching, stopping and fixing the code when a model gets stuck.
This still lets me stay engaged, and to modify code to be maintainable etc…
It gets me 80% there and I use to keep a subscription but often times just using googles AI mode is just as good.
That said I have 30 years of experience and insist on knowing how my code works, so this gets me 80% of the short term benefits while not depending on a 3rd party to keep my code moving forward.
Your mileage will vary and 2*5060ti 16gb cards would get around 100/tps with Qwen 3.6 35B-A3B on cards that are widely available.
To be honest the more modern cloud models are using draft tokens etc… that while they are superior for common coding tasks are degrading with more domain specific tasks.
That is just the cost of the draft model being ~10-20% of the foundation models size, and even the biggest Blackwell GPU is limited to ~250/tps so MoE or draft models are required for scaling performance at the foundational level IMHO.
The hard part is my use case are the OOD or small examples in corpus level, the above hurts there.
A Lamborghini may be nice, but I personally need a minivan more.
Quantization + KV cache paging + speculative decoding (MPT or draft) is a fairly good mixture here.
Some examples as I don't have access to run tests on a 5060ti right now:
https://njannasch.dev/blog/gemma-4-mtp-vs-qwen-speculative-decoding-5060ti/#vs-qwen-36-mtp
https://www.reddit.com/r/LocalLLM/comments/1umw7vj/dual_5060_ti_16_gb_llm_inference_performance/
27.21.533.298 I slot print_timing: id 0 | task 7843 | n_decoded = 1780, tg = 62.15 t/s
27.24.537.173 I slot print_timing: id 0 | task 7843 | n_decoded = 1965, tg = 62.10 t/s
27.27.541.142 I slot print_timing: id 0 | task 7843 | n_decoded = 2152, tg = 62.11 t/s
At 10k context I get about 40 tps generation and 500 tps prefill. At 100k context I get about 25 tps generation and 400 tps prefill.
It works, but I often use gpt or claude to make a detailed enumerated plan of what I want to do first, then have qwen follow it.
I'm not sure if it is economical or not, but I have solar on the roof so the power use is not really an issue and I already have the hardware.
The biggest benefit for me is that it's all done locally, and I know the harness is not uploading anything or sending telemetry to someone else.
Are there any articles you’d recommend for this?
I have Qwen running on an HP Z8. Very nice platform.
I have mine in a sandbox, due to privacy fears.
Your solution sounds more elegant.
I really just iterated over the harness over and over for about two weeks with opencode until I was sort of satisfied (still lots to do there :).
For the llama.cpp I asked claude fable to optimize it for my hardware and iterated a few times. In the end I landed on the following: https://pastebin.com/2PpJFUC0
The flip side is that it takes four H200s to run, and that will only let you cache context for maybe three users.
Fingers crossed our Blackwells show up and Kimi 3 really releases weights, because at some point devs are spending a significant portion of their salary on tokens and it's somehow cheaper to buy these ridiculous DGX servers and rack and run them.
Can't comment on how it compares to plans (I really don't like the limitations and general shenanigans I see around plans, so I've never tried them).
It is notably slower than Fable / Opus / Gemini, but also vastly cheaper than their API pricing.
Deepseek is at least on par with Sonnet (ghcopilot at work)- I don’t use opus, too spendy and I don’t need that level of ability.
The cost is for me was $5/6 months of use. Not a big user I guess! It’s good though, fast enough and incredibly inexpensive.
Been testing Qwen 3.6 28B on a 5090, and it’s also quite good for “free”.
I mostly do small self serving embedded projects based on esp32, so not very complex.
I think that's the case for people who compare it to proprietary models paid via API—which I think is irrelevant given the majority of people daily driving AI coding are doing on a subscription plan.
The better analysis then is not about AI coding, since there's no subscription plan for Kimi K3.
Instead, compare the cost of running some agentic _task_ that isn't coding which can only be done via API. Think of all the startups wrapping around ChatGPT and Claude to provide some additional set of tools, context, data and hoping to turn it into a profitable service.
To those companies, which are many, open models are the difference between the math working out today vs. praygeing they can scale fast enough to find profitability.
Claude and ChatGPT are good deals right now, with the subsidies. They produce things faster and better. I guess not cheaper, in that inferrence on my macbook is basically free, although the macbook itself definitely wasn't. My focus on running local is around three principles:
1. I don't want to support surveilance capitalism by giving these companies my data anymore, when I can avoid it. And LLM companies want to vacuum up every detail of your life.
2. I don't find these companies to be remotely trustworthy, and I find them hostile to a healthy society, so I want to avoid giving them money going forward
3. I think they're going to start charging a lot more
As I see it, an investment in AI hardware is an investment in my own future.
IE, I drive my car a couple of days a week, and it’s perfectly normal to spend $500 a month on an asset like that. When you factor in the SPACE it takes up, that’s the REAL cost of owning a car: the real estate you have to buy for your car to occupy.
Once that’s factored in, the “true” cost of having a car can easily be $2000 a month, even for a crummy car. The space that the car occupies is expensive.
Yet people balk at spending even $2000 on a GPU.
Makes no sense to me. I choose to invest in the future.
When I expect to need a lot of tokens and the task isn't too difficult I use sota to plan and create a thorough set of instructions and let deepseek chip away at it. With thorough instructions the quality tends to be satisfactory, and you pay something silly like $15 for 600m tokens.
GLM 5.2 seems like a decent price/perf and Kimi 3 has some real nice performance for an open weights model, but gpt 5.6 is unexpectedly affordable (especially if you don't automatically use Sol at max) so I don't think either is worth it atm. The exception is when you're working on something that US models get cold feet about, which seems like a constantly growing list. For me Fable is already too much of a headache in this regard, but chatgpt is still okay-ish. Hopefully it'll last. If not, there's Kimi.
tldr SOTA for most things because gpt 5.6 is token efficient. If I expect to burn a lot of tokens I use deepseek 4.
It's not as good as the frontier models I use at work, but it's plenty capable for the types of tasks I am using it for.