The point of local models is privacy, offline use, and maybe no guard rails.
* Not talking about enterprises that buy DGX racks and host Chinese models for internal use.
But also, control and consistency. A local model cannot be changed out under your feet like an API model can be.
Sounds like we have similar boxes - mine has a 10 core CPU, 64 GB of ram, and a 2070 Super. My motherboard had two unused PCIe3x8 slots and doesn't support Blackwell GPUs. I bought a couple of brand new Ada generation RTX 2000s with 16GB of memory for under $1400 to get to 40 GB of VRAM. That will easily run Qwen3.6-27b at a 6-bit quantization and 80,000 token context size. It isn't fast (19-21 t/s), but using pi-coding-agent is fine.
Now, my instinct is that I am giving up SOTA performance on agentic coding with this setup and LLM. But the gap between my setup and SOTA commercial models is small enough that it doesn't matter to me.
We all know that is hugely subsidized, and I guarantee that OpenAI and Anthropic are looking to enshittify that ASAP.
The enterprise users, however, are not subsidized like that. They pay per token. And some developers in those companies are chewing down a lot of tokens. Self-hosting an open weight model could be a massive savings very quickly. It also gives them negotiation leverage when talking to OpenAI and Anthropic.
For all we know, inference might be dirt cheap, they might just be hiking the API prices so high for us to think subscriptions are subsidized.
Now, the one wildcard in all of this could be Google. They are on the eighth generation of their TPU and have been holding their cards extremely close to the vest. I don't think anybody has a good read on exactly how much capacity they have. Most things you can kind of figure out the overall business numbers and what's going on in Google--the TPU area is one of the exceptions. I know a couple of big customers and even they don't have any visibility on that front.