You'll spend ~100x more on electricity than the API cost to have it run on someone else's GPU at several hundred tokens per second.
I think some sort of extreme data privacy requirement is the only situation that justifies this, but the intersection of {needs absolute data privacy, needs to run SOTA model, cannot afford GPUs} is really really narrow. I wouldn't be surprised if this is an empty set.
There are parts of states like Grant County Washington that have cheap hydro power, but it's very rare for power to be that cheap in the US. Even if this applies to you, it won't apply to the vast majority of people on here who will have electric rates 2-4x higher.
Average electric rates by region:
New England 28.1 cents
Mid Atlantic 25.1 cents
East North Central 20.8 cents
West North Central 14.8 cents
South Atlantic 16.1 cents
East South Central 15.5 cents
Mountain 14.6 cents
Pacific Contiguous 26.1 cents
Pacific Noncontiguous 42.1 cents
I think you can get down to around 8 if you are signed up for an interruptible load, or a dedicated off peak load, depending on the company, but yeah, standard rates aren't that low.
This is a bit misleading, because it's combining the 50 cents/kWh from California with 15ish cents/kWh in Oregon and Washington. Seattle City Light, for example, charges 13.38 cents/kWh on flat rate pricing, and far less with time-of-use billing (8 cents/kWh on off-peak).
From my last bill
> KWH USAGE 2590 - $183.37
There's a base customer cost of $18 on top of that, but yeah ~$0.077/kWh taxes included.
Last one was $212 for 2,146 kwh between June 8 - July 6 (28 Days)
Using myself as an example:
I adjust my A/C to run outside of 5pm-9pm (peak) if at all possible, we try to avoid pointless high-draw usage during that same window, and both of our EVs hold off charging until after 9pm.
My rate from 5pm-9pm is 0.43/kWh. My rate after 9pm is 0.09/kWh. The flat rate alternative, if I did not want to worry about time of day, would be 0.21/kWh. These prices are all-in, including transmission and distribution/whatever.
It would be dishonest to say that my EVs only cost me 0.09/kWh to operate, which on it's face is a claim to paying over 50% less. In reality, time of day pricing typically saves me somewhere between 10% and 15% in an average month compared with flat rate.
You can do the same thought experiment with say a dehumidifier in your basement. It can easily be off during peak usage and still accomplish its job, so its cost of electricity is also the marginal off-peak rate.
It would cost me more (modestly so, less than 10%) to be on TOD without the EVs. This will vary by customer, of course, and I expect that the power company designs TOD to be a wash for the average customer. They even guarantee it won't be more than 10% more expensive over the first year or they will refund the difference.
I guess it depends on if you would be using ToU otherwise.
It looks like about 50% of Californians use ToU plans, but the number is only 10% nation-wide.
https://www.hydroquebec.com/residential/customer-space/rates...
Another example would be Manitoba hydro
All figures in Canadian currency
Around here electricity companies quote prices like yours but that is supply only while transmission, taxes, and fees are again as much on top. Is that really all inclusive?
People should always compromise speed for data sovereignty! Who said: that in this digital day and age, information about money is more important than money!
1. Their API server provide an attestation JWT. This JWT is signed by Google's private key. 2. The attestation has details on the running container. I suppose the container host is a Google-provided distro and Google's signer will verify that the OS is theirs and up-to-date. 3. They could've proxy the attestation. To prove this is not the case, the field eat_nonce include the TLS certificate fingerprint, which should match the API server you're connecting to. I suppose you will need to pull their container and verify from the source that the container itself generate the private key, it never leaves the container, and the container has no way to run arbitrary code such as SSH or vulnerabilities.
Do you actually need to run the state of art model at 5 tokens per second instead of a qwen or whatever 7b or 30b model at 100 tokens per second?
On the other hand, would it be cool to also have a really big thing as an ancillary tool that I could throw a request into opencode before going to bed, let it crank away and take a look at what it's done 7 hours later? Yeah, particularly if I (very much an unknown quantity at this time) could be confident that it builds high quality, syntax valid, appropriately commented and not absurd code.
Some people like doing things they want to do. Do I actually need to buy expensive pigments from europe to make paintings of flowers? My camera produces a much more accurate representation.
a) Contracting with a third-party independent inference provider who will run your choice of model on fast hardware that they own, with all appropriate data security/privacy/contractual/compliance protection in place
or
b) Contracting with the original creators of the model to run inference via their API and with assurances that all the same data protection is in place
or
c) Spending the money to buy your own inference hardware to run it on something you fully own/control at proper usable speeds?
Edit: Everything I've been writing in this thread is mostly within the context of being able to evaluate K3 and its usefulness to be self-hosted as a preliminary proof of concept or test of feasibility of a new thing, such as on <$20,000 of server hardware, before proceeding to spend 300-400k on GPU-related hardware, or external third party services/ongoing billing.
They'll give you HIPAA compliance, they even have a data center for US government classified data, they can give you European data sovereignty. And with OpenAI and Anthropic models to boot, you don't even have to settle for open weights.
What kind of privacy needs do you really have beyond that?
Even for EU companies running open weights on EU stacks LLM inference on the GPU must process plaintext and I can't find any EU provider with NVIDIA H100/H200/Blackwell CC mode plus SEV-SNP or TDX, where you can cryptographically verify the workload ran somewhere the operator cannot inspect.
Personal compute is therefore the only option if you want personal autonomy privacy for IP &c. Maybe another option is to use cloud compute rented to fine tune a personal model that suits your own needs that would help bring the cost down, I don't know enough about this area to know if it kills the "intelligence" of those domains due to limited ?cross-verification within the LLM.
for anyone not US-based, this company is hostile and you have to assume the US government can and will force them to give access to your data.
A trillion-dollar business can easily trade dollars for the privacy. A business with $1M to spend won't even get a phone call with OpenAI or Anthropic, who were the only* previous players in town for doing this.
Worst-case example: Bootstrapped startup working in military.
It's also the case that an open model enables many more intermediate-cost solutions. E.g. providers certified for specific applications, on-prem rentals, etc.
* Omitting Azure, which gives some privacy for some $$$ on their models, but not at the level of high-security.
That's the easiest case.
AWS Bedrock models running in AWS Secret Cloud for Industry. (I really have no affiliation with them, I'm just like... this is a completely solved problem, why do people think this is hard and requires on-prem hardware?)
https://www.aboutamazon.com/news/aws/aws-secret-cloud-for-in...
I'm with GP that these are tinfoil hat concerns, when there are solutions to all of these, unless you're perhaps in some country with very specific needs beyond things like European sovereignty or US military secrets (like a non-US defense concern).
Note that the other commenter never said US-based military oriented startup. You just assumed, then jumped to "heck yeah let's use Amazon Secret Cloud for Industry"
Not everyone has or wants an office in Crystal City.
If I were ranking third parties on their ability to safely handle my data without compromising it, I would rank Anthropic pretty low for things like Fable (where they more or less promise that they will misuse my data), but I want Azure pretty low in the sense that I fully expect them to be compromised.
I would tend to trust Amazon to avoid being compromised.
The provider needs to comply with specific rules, have specific certifications, and sign specific agreements. You check the boxes, and you're good to go.
Microsoft does that better than anyone. OpenAI and Anthropic don't do that at all. Google does that rarely and poorly. AWS is not bad, but not as good as Microsoft.
Azure was always my go-to for regulated applications in the cloud. Some do require e.g. on-prem or even air gap, where even Azure is out.
I have some inference I simply don't want to run on OAI, Anthropic, or Google because I don't want to run afoul of their "rules" and end up with a banned account, and this situation is only getting worse when it comes to doing fairly basic tasks like trying to secure your app against security problems.
Qwen 3.6 is another matter. Paying provider rates for the amount I run locally would put me in the thousands of dollars. So that's very practical to buy a Macbook instead, plus an RTX card, and so on.
Are there? At the highest levels of defense and law, AWS and Azure are used.
Having tried selling some of these entities on doing things in-house, there seems to be little interest.
This is certainly true if the user is an American company. You could look at the European initiatives to run this stuff on hardware they own in facilities they own and control within the borders of Europe for a counter-example.
Such as: https://www.google.com/search?client=firefox-b-d&q=schwarz+s...
https://www.dutchnews.nl/2026/04/government-turns-to-german-...
Hopefully that changes!
I don’t know if that’s 100x more than I’d pay (opex-wise) with an nvidia setup, but I can say the one-time capex is a great deal cheaper. Avoiding VRAM and DDR5 (fast DDR4 should be OK) are the biggest cost savers. ECC RAM is worth the extra price. General datacenter-quality hardware has less price sensitivity, and plenty of bang for your buck.
Under heavy inference load you will find that the cpu usage is actually less as the bottleneck is the RAM bus speed. An older 2U rack server that is 600W load (typically a 1+1 power supply server when plugged into two kill-a-watt would show 300W on each, equal load balancing) when maxed out with stress-ng might be only 450W total running inference.
If you have 600kWh used in a month by running something 24x7 and your power is $0.15 a kWh, that's more like $90/mo (not counting cooling or any ancillary costs for the environment where it's in).
The options for AC power supplies for servers with 2 or 4 load sharing redundant power supplies are a lot greater. If you were buying all new hardware and starting from a clean sheet of paper design with lots of money to spend, absolutely. At that point also start looking at higher voltage DC distribution stuff related to open compute platform and 800VDC.
But if I were trying to make the absolute most use of $20,000 to put together a 3TB RAM server (48 x 64GB DIMMs), it would likely end up AC powered.
Where in the world are you finding that much RAM in a racked server for $200/month?
That's a world that I don't think we're ready for.
Young men 14-?? already compromise and attempt to extort organizations daily, sometimes cluelessly from western nations, often not. It doesn’t have to be gangs when the home country doesn’t care / isn’t technologically or culturally developed.
Already seeing AI-written payloads and frameworks in the wild. I think it’ll turn out that AI won’t build you a maintainable ERP but it can create C2 networks, exploit POCs or even 0-days potentially, and let kids make their own ransomware tooling. Then we’re dealing not with a handful of cybercrime tool makers but a generational problem.
Don't get me wrong, slow is sometimes better than "not at all", but depending on the performance, it might end up way too slow to even work for batched/async jobs like that.
And then that's just for single prompts, what about agent harnesses, where before every tool call the model could reason a bunch?
I agree with you that real world results would be interesting, but I wouldn't hold my breath nor expect it to realistically be able to be useful. Still, people should try it, for science if nothing else :)
So full CPU local AI inference may become viable option in coming years.
Or from SSD using something like Colibri[1]. Not going to be quick, but at least runable.
edit: the results I have seen from people trying colibri with fast consumer grade PCI-E 4.0 NVME SSD are 0.1 tok/s on models that are <700B in size, things that are well under 800GB on disk. With something that's 3T in size it'll probably be a lot slower than hat.
But still fun you can run it at home.
The model is known to be MXFP4 according to Kimi's release blog post, so the model weights will be less than 1536GB: https://www.kimi.com/blog/kimi-k3
Also, their previous models were native INT4, so it would be weird if they went larger now.
* Sparse Experts: 1481.4 GB
* Dense Experts: 1.9 GB
* Self-Attention: 72.4 GB
* LLM Head: 2.4 GB
* Embeddings: 2.4 GB
* Vision Encoder: 0.35 GB (surprisingly small)
plus some miscellaneous parameters.
Most importantly, we now know that the model has 104B active parameters, which is quite a lot and will make it difficult to self-host efficiently.
On a 3T model I’d imagine you’d be closer to 0.05 tks
If wonder if you can train a model to optimize this, by trying to make the expert selection sticky across a few tokens, without too much quality loss.
Another fun idea might be to try to build a model where the router chooses the expert 1-3 tokens in advance.
You can!
> AFM 3 Core Advanced makes routing decisions per prompt. A lightweight, dense block selects a fixed set of experts during initial processing, periodically reselecting them during generation.
https://machinelearning.apple.com/research/introducing-third...
But the memory bus speed is fully committed when generating tokens or thinking.