With an 8x MI355x cluster at full tilt and including cooling, your power draw runs ~17kW. That's what it looks like when it's running full tilt. To be fair, hey that's pretty expensive. It does mean 8 multi-trillion parameter models unquantized running 24/7 without pause. And you get the full month like that, your monthly token limit is the time in a month. That cluster, the electrical upgrade, the cooling setup, and the electricity to run it all costs less in 2 months than your maximum affordance from Anthropic does in the same time period. Two billing cycles, and realistically it's more like two weeks. In 4 quarters you've wasted over a million. Like, what are we talking about here?
Now if you aren't using AI all that much, which is perfectly valid, and especially if you aren't using it at its absolute maximum, the story changes. Because even though at that point you're not paying nearly as much in electricity to run the cluster anymore, you still have the $300k+ capex to get the setup in the first place. But if we're not redlining it non-stop, then we're not really talking about performance anymore, are we? If your org never comes close to hitting token limits, it's probably because AI is rather marginal for you. Which again, is perfectly valid. I don't even use AI professionally.
Fact of the matter is, if your corp can justify the capex for a cluster and makes heavy use of AI, you are literally burning money by not having one in your building. The numbers are painfully obvious. Even deepseek isn't as cheap. This is before we get into things like LoRAs, custom inference pipelines, etc. which you know are kind of important if you actually care about model performance.
If you just ask "who spent more in the first year" (100% depreciation) then even with 5-6 max accounts, buying HW will be a couple of times more expensive. But when does it make sense to ask that question?
Maybe the SotA models will need better hardware so your investment will not be useful after a year or you'd need very expensive upgrades? But then (as in Fable case) subscribers need to spend more too.
Idk where you live, but where I am running the M5 Ultra Mac Studio at max rated power 24/7 for a month costs C$42.
The considerations against Apple hardware are 1) hardware advancements 2) early access to the best models. But it’s really not that clear.
(The other guy who thought hosted models on openrouter are cheap has spent $100k in 5 years.)
Pretty expensive is an understatement. You couldn’t buy one of these if you wanted to right now. If you could it would be multiple hundreds of thousands of dollars.
> It does mean 8 multi-trillion parameter models unquantized running 24/7 without pause
You can’t even run one unquantized multi-trillion parameter (>=2T) model on 8 x MI355x with enough context for concurrent users. I don’t know how you think it’s going to run 8 of them at the same time. Did you mean 8 concurrent sessions?
Your math is way off across this post. If replacing an Anthropic subscription for a whole company was as easy as buying a box for the office and then breaking even in 2 months, it wouldn’t be some little secret that we only discover in a comment online.
You can: https://www.exxactcorp.com/Exxact-TS4-149591758-E149591758 . You can get thousands of tps of GLM 5.3 output out of this thing, which grades around Opus 4.8. Payoff is around 1 year vs. spot prices on these GPUs, including power.
No, you can get a quote for possibly being allocated one in the distant future.
The backlog for these is huge. You cannot buy one any time soon.
They're a good provider but you have to be a big shot buying NVL72s before you're getting anything within your payback period.
Also, the lead time I quoted was for individual 8x nodes.
A system with 4x RTX 6000s costs about $60K these days, and can (as you note) trade blows with Opus 4.8 if not Fable. In fact, it'll give you a better pelican than Fable 5.1, and in less time.
Okay I love the open models, but the hype is getting ridiculous. The models you can run on 4 X RTX6000 are not Fable level.
And Opus is no slouch. I'm satisfied that GLM 5.3 is just as strong as Opus. Z.AI has promised/bragged that they will be at Fable 5.0 level by the end of the year or early next year, and I don't see any reason to doubt them.
Obviously, I quantified both the operating expense and the capital expense in my post. What I find curious is that you're quoting me talking about the operating expenditure, and changing the topic to be about the buy-in like these are interchangeable things. You don't think that this is a crucial and important distinction?
> You couldn’t buy one of these if you wanted to right now.
You could have spent all of 5 seconds of searching rather than just assuming[1]. You're not buying an Nvidia Superpod™.
> You can’t even run one unquantized multi-trillion parameter (>=2T) model on 8 x MI355x with enough context for concurrent users.
That's certainly fair a point. Although in the English language, especially in legal contexts, the multi- prefix is used inclusively for fractional values. That is it's strictly >1, not >=2. IE an 18 month contract is a multi-year contract, or a $1.6 million dollar asset is a "multi-million" dollar asset. But this is uninteresting semantics.
You are right, but it also doesn't matter. The gap is just that big. You can run 1 single user of Kimi K3 and still not even come remotely close to the $70k or so that a single Opus 4.8 user can burn over the course of a month on left on max. An honestly lowballed amount I know from anecdote. The per-token cost is just really expensive.
> Your math is way off across this post.
You made one technical point above, one that doesn't ever arrive at a relevant rebuttal to the substance of my post. But please, I'd love to hear you elaborate, especially because I didn't actually give much math at all.
If you want math though, here's the math. Let's say you are paying a ridiculous amount of money for electricity, a price nobody in the US pays -- $2 per kilowatt hour. That's about 5x the average rate in California, 4x as in Hawai'i. 17kW @ $2/kWh * ~8766 hours in a year puts that cluster's electrical costs at just shy of ~$298k annually assuming it takes no breaks. Let's make matters worse and round that up to $300k. It's also assuming you didn't invest in a solar hookup for your building, which I don't know why you haven't at this point, especially if you're installing a CDU for your new cluster. 12 months of Claude burning $70k a month is $840k. For a buy in of, you know what, let's call it $500k. Why not? It still doesn't matter. The operating cost is so much lower it's paid for itself plus an additional $40k in the first year. Even at a ridiculous penalty in electricity that nobody pays, even overinflating the amount of money you'd pay for the cluster and the infrastructure to get it set up, it's not even remotely close for a single user where the gap is smaller (IE, you're not wasting "a million dollars" in a year by maxing out the $200k scaling limit every month)
You can of course trot out the point that oh, in 12 months this setup will be extremely outdated! It doesn't matter. If the work it was doing today was useful, it will be useful next year too. And with the rapidly encroaching diminishing returns from parameter scaling, you're probably going to be just fine for a while. Maybe grab a quantized version of a newer Chinese model at the end, before grabbing a newer generation of AMD node. Those MI400s are looking pretty sweet after all.
> If replacing an Anthropic subscription for a whole company was as easy as buying a box for the office and then breaking even in 2 months
If you're locked in, then you're locked in. But don't pretend like you're saving money. You're not.
> it wouldn’t be some little secret that we only discover in a comment online.
Why does this have you so nasty and defensive? It's not a "little secret" that running your own infrastructure is cheaper. Of course it is. You know what else is cheaper? Owning your own office building out in the sticks, rather than leasing part of one in the city. Not everybody can make that work, there are no free lunches after all.
History repeats, these same exact lines were rolled out ad nauseum during the cloud craze. Datacenters are businesses, not charities. Frontier companies rent quite a fair amount of their infrastructure. Even if they resold that compute below cost (they don't), there's a pretty steep cliff before the economics start to look attractive.
[1] - https://www.avadirect.com/GIGABYTE-G893-ZX1-AAX4-Dual-AMD-EP...
99% of the cost was in input tokens, I only used like 100k ish output tokens. It was a one shot task asking the agent to implement proxy injection to Guice. It did a pretty amazing job.
If you were to use hosted LLMs for a lot of agentic coding, a maxed out M5 Ultra Mac Studio would pay for itself in under a year.
Considering that I hit the 1M compaction multiple times per day with codex, it would definitely cost at least $5-8/day to use deepseek how I normally use codex.
He was a lead engineer, so after he announced it wasn't going to work, everyone pretended it never happened. But we all knew.