1. Insanely discounted starter plans. Claude $200/mo plan is like $5k-$8k of API rate usage.
2. Very limited cost visibility, they make it hard to figure out where you spent money (unless you're on the enterprise plan which is for people with unlimited money).
3. Nobody, not even the model provider, knows what your request will cost before it returns. You're writing a blank check every time you hit enter.
4. When you run out you run out very suddenly and disruptively. It's very hard to tell a developer on the 28th of the month "sorry, code by hand until the 1st of next month" so you tend to grant exceptions.
5. The price is changing all the time. New models come in, old models come out, prices change, caching behavior changes, harnesses change, etc. The cost of doing a single task is not predictable even if the task does not change.
6. Basically no volume discounting. Anthropic offered us 2% off for committing to $1M+ per year at API rates.
I manage AI spend for my team at work and I try really hard to keep costs under control but it's absolutely herding cats. Much harder than any other spending I've ever had to manage at work.
Happy to give you (or anyone here) some trial credits if interested! Email address in my profile.
In addition, we're seeing people applying AI to more and more use cases, so token growth is very significant. Paired with consumption pricing, it's brought this problem to the forefront very quickly for lots of companies.
It is true that this problem can be mostly managed by the techniques we mention here. Those are actually pretty difficult to set up at scale, so many companies (including us) we only really did this in earnest once we started to see those large cost oscillations.
The main reason we shared this here is to maybe help other companies get infrastructure in place before massive cost swings rather than after.
What didn't happen is any analysis of cost/benefit up front. Many of the corporate decisions around AI have seemed characterized by companies blindly copying each other.
This is how AWS made its fortune.
Initially, you picked the Multi-ZA RDS db.t3.2xlarge instance because you figured "eh i have credits anyway". Two years later, someone looks at this and says "hey, this is expensive and I bet we can do everything we need on a machine half the size". But then they think "if i downsize it and that works, i'll get a thumbs up emoji on a slack thread. If i downsize it and it causes problems, i'll draw the ire of the whole team. I better leave it alone." And the truth is... by the time your company hits the end of those credits, you're probably at the point where that savings isn't gonna do much. Or maybe you are out of business.
And that is how almost every successful company that uses AWS eventually ends up paying six-figures or more annually.
They’re not saying they regret doing it, or that it was a mistake.
They’re just saying they’ve gained experience and have leveraged the tools to an extent their usage can be optimized.
Pretty standard business or life iteration.