They have infinite money to burn so they aren't going to optimize.
I've seen that trivial changes to workflows can token usage by 30%. Repeat this a few times and my token usage has plummeted over the last few months.
Only outputting test failures instead of having agents read test run output in real time. Not having build logs get fed into context. Projects like UCL (unified context layer) so agents don't rediscover the same facts about a code base again and again.
Giving tools agent output modes so they only print out the minimal info needed for an agent to get work done.
When I hear a company has an absurd AI spend I assume they aren't even trying to reign it in.
IPOs aren't the magical moment that they were in decades past. You get to answer to a bunch of retirement fund managers and "the market", meaning less-savvy retail investors. You have to open up your books.
It seems like Anthropic, OpenAI, etc. are doing an IPO because they are literally out of other options. Older IPOs like Facebook felt different, like the hard part had been done, and they just had to scale. There's a difference in selling something because you want to, versus selling something because you have to. Facebook was just relying on the idea that they were a platform that more people would inevitably sign up for. For the most part, they did. It seemed obvious. They likely could have continued on by simply operating privately, but the IPO gave them capital to dominate the market.
Anthropic and other hyperscalers cannot continue on privately. They're facing a systemic shortage of capital to achieve their goals. They need the infrastructure, and there's not enough capacity to make enough for all of these companies to survive.
It’s not opportunity cost. It would be opportunity cost if their employees not using it would lead to additional revenue. However, if Inwas to go to Anthropic and double my token usage Anthropic wouldn’t decline my money. If Anthropic’s employees would stop using Claude, their revenues wouldn ’t increase by an equivalent amount.
Their revenues are not currently supply constrained.
It’s just a cost cost. If their employees stopped using these services they would simply need less infrastructure, and therefore save on the cost to build and run that infrastructure. It wouldn’t translate into an increase in revenue.
I don’t agree. This is determined by elasticity of demand. If they have more compute to offer, that means they could offer higher subscription limits, more features like Duo to more people, better API pricing, and better enterprise contracts. This leads to some degree of increased consumption and more revenue. The degree is determined by how responsive customers are to pricing and features and performance. It’s not zero, so there is some amount that revenue would increase. We can debate the coefficient and the amount of compute we’re talking about here, but the premise is correct.
you could not be more wrong, you cannot IPO if you say product is good enough. The whole point, the bet why its worth trillions is because they will reach AGI and "singularity" first, so the gap vs open source will only increase. If you say product is great as is, then you will be caught up and game over
In order to correctly make that decision, they must use the opportunity cost of selling that token. I'm pushing back at OPs claim that Anthropic must use internal cost to calculate that trade-off. They absolutely should use external selling price.
They are doing the calculation on if the $150K/person is worth it, and it probably is.
Their revenues have exploded, they are unit-profitable, they need to also invest in the future.
AI infrastructure is supply constrained, so you have to make a decision: how much do I allocate to training vs inference?
Do I use it all for inference and do no further training? If so, what risks does that expose me to?
It's supply constrained in terms of global hardware production capacity, but that's not how it works for any given company. If Anthropic wanted more GPUs for customers paying them enough money to outbid the next guy in the market for the finite number of GPUs then they would do that and get a higher percentage of the finite number of GPUs.
So the cost of using more internally is still their own cost, i.e. the price of being the high bidder on more compute resources, not the retail price they charge to customers.
That's API-list prices. They have an 80% margin over COGS.
Cost per equivalent output is dropping 10x every 18 months -> so in 3-ish years, if the trend continues, that will be $3k per employee.
Not bad. A lot of companies are already spending about that much or more on IT per employee.
Especially since that $3k bakes in an 80% margin. It's about $600 per year if you DIY at scale - which is about $50 per month - but at the $3k price, hardly anyone is going to.
In the West old-age dependency ratio (retirees per working-age adult) is climbing roughly 0.5 to 0.7 percentage points per year.
We've got to get 0.7% more productive for employees to keep the same quality of life (unless retirees get screwed, and since they vote and make up a huge portion of the voting block, I wouldn't count on that happening).
Further, more and more of the population has to transition into into elder care.
Unless we hit 3%+ GDP growth, us workers are probably going to see declines in quality of life (and we're probably not going to get those levels of growth)...
You could also look at the opportunity cost of not letting Deepmind do quite as much research. How much is additional research worth on the margin? There will be diminishing returns.
Bank reserves, Compute etc are trade-off decisions. Buffett vs Mayoshi Anthropic vs Google are being played out in real-time for us. Pretty soon you'll know which trade-off was better.
source? The most optimistic calculations don't come anywhere close to that.
They have never claimed that the API rates are the "real cost" of the compute spend.
I don't know where you got that from but the only time I maxed my $20 plan for a week I made the calculation that I had spend close to $200 in a single week. So maxing a $200 plan for a month should give you $8000 worth of token at API price. Obviously it's not $300k but it's still very far from the retail price.
$200 of usage on a $20/mo plan is a 10X multiplier.
The $200/month plan has 10X the usage of the $20/month plan.
So $200 * 10X = $2000/month. I don't know how you got to $8000.
Recent analysis by industry analysts like Semi Accurate have measured Sol 6.1 on the $200/month plan to be around $2000 of API equivalent use.
FYI there are 4 weeks in a month…
> Recent analysis by industry analysts like Semi Accurate have measured Sol 6.1 on the $200/month plan to be around $2000 of API equivalent use.
Sol isn't an Anthropic product though so maybe OpenAI is less generous than the former, I can't tell.
Thats a problem since they will always pay a pemium for compute. The data centers owners can compete on price if they choose to.
You have confused necessary and sufficient.
Maybe, but it also reduces supply that can drive up prices
There are a bunch of open source harnesses that are competitive with CC and they aren't using anywhere near that many tokens to do it.
Anthropic recently handed out free cloud resources (it's still free, guess they want you to get used to the fact that cloud is not free). I went through $250 cloud budget in less than 2 days.
Feels almost as expensive as LLM calls.
However, so is my phone!
Even within a single moment, if there is capacity for an internal request, just send it, if market traffic is currently at 100%, don't.
This is NOT the definition of opportunity cost. Please do not throw in random terms you don't understand. Opportunity cost is the value of the next best alternative you give up whenever you make a choice. WTF does it have to do with mis-classifying an expense?