undefined

points

[-]

It depends how we are looking at the business. Absolutely at the end of the day a company is profitable or not but when thinking about inference, which is largely a commodity these days, you would first think about the marginal cost of it. That is your corner stone of the business. We have pretty clear indication that largely API tokens are being sold above the marginal cost. For especially a brand new business that’s critical and something that many unicorns never even hit.

Your right that all other costs are critical to measuring the profitability of the business but for such a young industry that’s the unknown. Does training get cheaper do we hit a theoretical limit on training. Are there further optimizations to be had.

You don’t have large capex in an industrial and then in year one argue that the business is doomed when your selling the product above the marginal cost but you have not recouped costs yet that have been capitalized.

by overrun111 hours ago|

prev|

[-]

The article is about compute cost though. By "lose money on inference" I mean the assertion that inference has negative gross margins which a lot of people truly believe. This is important because it's common to reason from this that LLM's are uneconomical and a ticking time bomb where prices will have to be jacked up several orders of magnitude just to cover the compute used for the tokens.

by mike_hearn51 minutes ago|

parent|

[-]

But there's no such thing as compute cost in the abstract. What exactly is compute cost for AI? Does it include:

• Inference used for training? Modern training pipelines aren't just gradient descent, there's a ton of inference used in them too.

• Gradient descent itself?

• The CPUs and disks storing and managing the datasets?

• The web servers?

• The people paid to swap out failed components at the dc?

Let's say you try and define it to mean the same as unit economics - what does it cost you to add an additional customer vs what they bring in. There's still no way to do this calculation. It's like trying to compute the unit economics of a software company. Sure, if you ignore all the R&D costs of building the software in the first place and all the R&D costs of staying competitive with new versions, then the unit economics look amazing, but there's still plenty of loss-making software startups in the world.

Unit economics are a useful heuristic for businesses where there aren't any meaningful base costs required to stay in the game because they let you think about setup costs separately. Manufacturing toys, private education, farming... lots of businesses where your costs are totally dominated by unit economics. AI isn't like that.

by jeremyjh35 minutes ago|

prev|

[-]

This is all true but it isn't really important for the argument people are making. What is more important is the marginal cost per token. If each token sold is at a marginal loss, their losses would scale with usage, that simply can't be happening with API pricing. But in general, yes I agree with you and I'm sure they are taking a huge loss on Claude Code.

by howmayiannoyyou2 hours ago|

prev|

[-]

You're missing costs.

- Amortized training costs.

- SG&A.

- Capex depreciation.

All the above impact profitability over various time horizons and have to rolled into present and projected P&L and cash flow analysis.

by ACCount3736 minutes ago|

parent|

[-]

We have amortized training cost estimates. Inference to training compute over model lifetime is 10:1 or over for major models at major providers.

In part due to base model reuse and all the tricks like distillation. But mainly, due to how much inference the big providers happen to sell.

So, not the massive economic loss you'd need to push models away from being profitable. Capex and R&D take the cake there.