undefined

points

[-]

I run qwen 27b at home when working it pulls around 400W. I get 40ish tokens per second generation and more importantly about 1000 tokens per second prompt processing.

In an hour it can process 3.6 million tokens or generate 144000 tokens. This costs me about 15 cents given my electricity prices.

For sonnet the equivalent token costs are 7.2 dollars for the prompt processing or 1.4 dollars for the generation. The cloud is 10x more expensive for generation and close to 50 times more expensive for processing.

by bischofs3 hours ago|

prev|

[-]

I don't pay anywhere near 0.30usd in the US - I pay half that off peak and can buy 1000$ worth of batteries to load up on super off peak (0.11usd). Also the inference providers are fighting over market share with huge debt loads so they are definitely going to go up in price.

by hagen82 hours ago|

parent|

[-]

Inference costs will go down massively once they use the upcoming GPUs. I estimated that a model like GLM5.2 will be around 0.03USD/M output tokens in 2 years when the Feynman GPUs will be available in 2028. And this did not even consider architectural efficiency improvements. In mid 2027 we will already see a 10x reduction once everyone has switched to the Ruby architecture.

It will be feasible for everyone to have 20 different agents running at all times. A new world is coming

by amanaplanacanal3 hours ago|

parent|

prev|

[-]

Yeah I had to check, I'm paying 0.08usd per kwh. This is in the US PNW with quite a bit of local hydro power.

by brewtide3 hours ago|

parent|

[-]

It's all relative. On the opposite coast, Maine it is ~ $0.28 cents kwh including getting it there. (~ 50% energy, 50% delivery). It's too darn expensive here.

by sosodev3 hours ago|

prev|

[-]

Of course efficiency matters, but a lot of people either have cheap electricity or efficient hardware. My AMD strix halo home server can serve Gemma4-26B at like 70 TPS (rough estimate, I don’t remember the exact speed buts its fast af) while only using 100W.

by carra1 hours ago|

prev|

[-]

> Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally.

Maybe, but for how long? Prices keep going up, and every new model eats more and more tokens...

by albrewer3 hours ago|

prev|

[-]

Not OP, but your math is a bit off - I have solar panels :)

by segmondy1 hours ago|

prev|

[-]

don't care, and yeah i don't like to hear it. we don't run local because it's cheaper money wise. we do it for freedom, for privacy and having option makes it cheaper in the long run. if there was no local options, your cloud model would cost much more!

by apitman2 hours ago|

prev|

[-]

It would be more efficient if you had multiple users (or agents) making parallel requests to take advantage of batching, right?

by kylemaxwell3 hours ago|

prev|

[-]

You pay 3x as much for electricity as I do, so the math here is going to work out very differently depending on a lot of factors.

by Waterluvian3 hours ago|

prev|

[-]

Locally I’m looking at about CAD $0.05 per kWh when off peak.

by newusertoday3 hours ago|

prev|

[-]

its 18k not 180k

by bigyabai3 hours ago|

prev|

[-]

It's the "Race-to-Idle" situation all over again. It consumes less power to complete a task faster, whereas using "low power" hardware that draws max TDP for 30 minutes isn't very power efficient.

The privacy nuts have a better leg to stand on, but even then it's hard to believe that they're using on-prem AI to replace SOTA model inference. As cool as local LLMs are, a lot of the stuff people run is a novelty.