- The publicly available information about how inference costs compare to training costs is conflicted. EEs involved in datacenters talk about power usage spikes during training runs as if they were a major factor in the designs, but academic papers discussing cost-optimal scaling confidently treat inference-time compute as a major factor.
- On the side of the balance indicating that training is more compute-intensive after amortization than inference is that Chinese providers, constrained primarily by access to compute, have nearly unlimited token availability at a lower price than US providers (inference), but poorer model capabilities (training). That would make sense only if US providers are inflating inference costs by 20-30x due to amortized training costs that overseas providers were not able to take on.
- If training >> inference, they're in a prisoner's dilemma that far exceeds the ordinary zero-marginals model of competition between firms (due to its huge discrete stepwise nature). On the other hand, if inference>>training, the high-level analysis popularized by certain thought leaders, that it's like a utility, would be true. You'd tend to count this as a vote for inference>>training, but the CEOs saying it at least have a huge incentive to agree because the alternative, the prisoner's dilemma, would stop investment very fast.
- The only voice in the story that I just told you to have anything to do with fact (as opposed to high-level analysis and ivory tower armchair management of a secretive business) were the rumors from facilities engineers. That shows you the state of our understanding...
- If we don't even know the ratio between amortized capital expenses and operational costs, outside investor analysis is impossible. It doesn't matter how finely they divide the accounting buckets for office ferns and indoor ferns if the single biggest part of their business is obscured for trade secret reasons.
Yes I know there's no evidence and this is lazy reasoning. But there's probably a bit of truth to this line of thought.
Speaking to your point, inference being dramatically less costly than training would not be seen as a delta from the norm. The things that thought leaders are saying, that they are providing inference for anything near the operational costs (like a utility would), is the delta from the norm.
Our estimated spend for AIaaS would exceed that cost in less than a year.
In a few years, there will be hardware capable of running frontier models good enough for most things at accessible prices for even tiny companies.
That's the game. There's a view you could take of this that this is just a growing of the pie: with those cost dynamics a lot more "small businesses" get a vast amount of leverage, so the overall economy grows without replacing the knowledge workers. I'm not sure I trust the MBA class to have that view.
I would argue that that's been the case for quite some time before AI. As an example, what innovative amazing world-changing products have Google or Meta launched in the past decade with their very high numbers of very talented and highly-compensated engineers? The issue with most big tech companies are leadership, strategy, and product direction. I'm not saying that they don't make any profits, just that they probably aren't "building [the right thing]".
AI for product development and management would be far more impactful than automating rote coding tasks / building React UIs that mirror API structures IMO.
I don't think there is any shortage of great ideas at these companies, they are just extremely bloated. And I don't think its something like indecision or bad PMs, it's "we have a finite amount of time and resources so we need to be conservative but also not too conservative"
If you have AI systems that can simply build out POCs in days, backtest on real data, show reliable results and numbers, you get a suite of product options you were never able to get before. If you have coding agents that can speed up implementation, you can build more stuff and choose the things that stick.
It changes the cost/benefit calculus of the entire business. I think you are exactly right in that: PMs/leadership are by their nature orchestration machines. Other roles are as well, but I think PM's are at a particular advantage here in that it will be quite awhile I would expect before core product decisions and creativity can be delegated to an AI, but not quite awhile until virtually everything that they're blocked on (legal approvals, POCs, wire frames, etc etc etc) will become less and less of a blocker
Yeah, if this stuff actually worked that well already, OpenAI et al. would just run AI CEOs and engineers. Why get some other company to pay you at all when you can automate every other company out of existence and take all the money they make?
The fact of the matter is that while the tech has some uses, it sure as hell isn't a full scale replacement and you almost always actually have to massage the input into LLMs to get anything decent back out in practice. Some CEOs and managers can learn to do this, of course, and some already are... but that quickly turns into a second full time job. A "programmer" is still needed. The job might change from mostly hand-writing C++/JS/Python to prompt engineering + some manual coding to fix all the stupid fuck-ups that the bots can't solve themselves, but you still need someone to actually prompt the bot.
When that changes, it won't just be engineers losing work; there will be no reason to even have a human CEO any more.
I suspect that AI will fail to pan out to the same extent for the same reason why outsourcing hasn't fully panned out (even though every company tries it after getting big enough).
The problems that will come up will be and always have been ongoing maintenance. AI is great at writing new code without a brain behind it, but once you get to the point where you need to refactor code, you start really needing someone with coding experience to guide the AI or veto it's mistakes.
I don't think that's really fixable even with a lot better AI. It's not something that ultimately comes out of the likes of github data.
I'm not saying that AI isn't going to make things better, btw, I just don't think we'll see a 20x improvement. Probably more like 1.5 or 2x.
The determinant of success was only whether the task needed American-tier labor or could make do with sub-American quality labor.
It sounds like the economy would largely reduce to the small minority class of independently wealthy people.
It takes a skilled knowledge worker to use these things.
Not completely, but compared to the middle ages we 50x'd their output. Which is a great illustration what it means to make a job 50 times more productive. We went from almost the entire population being required to make enough food to survive, to 4% of the population producing such an abundance that consuming too much food has become a systemic health issue
They do not care unless these companies can get a bailout.
UBI only exists for companies that are too big to fail. Case in point, 2008 and SVB when there was too much money on the line.
One of the AI companies attempted to guarantee themselves a way for the government to bail them out if they were close to defaulting on the debt from the data center build out.
Arguably, the main impact of securing SVB depositors above the $250k limit is that it prevented thousands of people from being laid off that week, as their employers wouldn't have had the money to make payroll the following Wednesday.
What makes you think the people who used to build (or would have built) software will switch into the industry of "knowing that the thing was the right thing to build", as opposed to something cooler like surgery, city planning or experimental physics? The roles within a tech company are not the only jobs in the world.
“There’s more capital than good ideas to fund” has been a complaint from the likes of A16z & other VCs for a long time now. It’s why we ended up with stuff like NFTs getting funded.
True, but I think the GP's point was that what consumers will pay won't be nearly as profitable as what enterprises will pay to increase the output of their developers and knowledge workers. ChatGPT is currently the overwhelming leader in consumer AI usage but only ~5% pay $20/mo.
As a recently retired technologist, I'm now one of those consumers. I use AI webchat daily for general search, Q&A and even to write little automation scripts for myself, yet I haven't paid anyone anything for AI yet. Even after being heavily restricted and performance nerfed in recent months, free webchat AI is still totally fine.
Even as AI compute costs fall over time, I doubt serving ads against AI webchat to consumers will generate the kind of high-margin, sustainable growth VCs get excited about. It's so undifferentiated I bounce around between all four leading providers because there's virtually no moat locking casual consumers to any chatbot beyond a single question thread. I guess if it had a nearly infinite context window integrated across all sessions, that might be somewhat sticky for consumers but it would also get creepy - and delivering that functionality is the most expensive thing in AI right now.
No one I know feels richer than they did a decade back. I've not been able to meaningfully put up my prices for a decade. People are tired and stressed and scared, particularly scared of a technology everyone keeps telling them will make them redundant.
There is no rising tide lifting all boats, just most of us drowning whilst a few whizz past in their yachts.
I honestly hope these guys faceplant ASAP. Couldn't happen to a nicer bunch of people.
Consumption has risen, inflation adjusted wages have risen for blue collar and white collar alike. Most social mobility has been the middle class moving into the upper middle class, not moving to the lower class.
The main thing holding people back is the housing crisis. This is orthogonal to the value creation of businesses.
Value creation is growth. If it didn’t exist the S&P would still be 42.55$.
What sort of new value, and why will people pay for it from someone else rather than prompting for it themselves?
They are assuming ~10% global GDP growth instead of ~3%. You probably don't need the same %s if the pie grows a ton.
I'm highly skeptical we get that growth, but if you aren't, it makes it easier to digest.
The more AI causes productivity increases, the less and less number of workers will be needed. This will heat up the job market even more and bring salaries down.
Net effect of this productivity increase: less consumption by the masses, even though you may be producing more good and much more efficiently.
A third effect also comes into play that once all this starts to happen, common people, who are generally living paycheck to paycheck, will now start to hesitate towards making any long term investment, housing included. And that indirectly will end up impacting financial and banking sector, which will then impact existing savings, bonds yields and retirement funds, and the recession-like cycle starts.
This productivity increase only makes sense if it is capped to a very small number.. like 20% max. Beyond that, who these companies will even be selling to?
Am I overthinking all this?
>Net effect of this productivity increase: less consumption by the masses, even though you may be producing more good and much more efficiently.
Big tech companies can't even create login flows and account recovery flows that work for everyone yet. There are countless stories of folks losing access to business Instagram accounts that get hacked, Google support from a human to fix a problem that is outside of their help articles is non-existent, etc etc. There's still so much "low-hanging fruit" IMO that isn't particularly fun or exciting to fix, but ask your average non-tech friend or family member what they think of the Facebook + Instagram security settings pages / sites / desktop-only settings.
Who is going to pay for all of these subscriptions that will power this GDP increase when average purchasing power of those outside of the top ~10% of earners is decreasing YoY? We're headed toward food and water shortages next to sprawling datacenters, not shared societal prosperity and a healthy middle class.
Secondarily, reducing the cost of making a thing doesn't always mean you get less of a thing. For me, certainly, what happened is that I write way more software than I originally did. When we built compilers, the amount of human engineering effort required to do things plunged, but the amount of software engineering jobs didn't go down.
This is as bad as models will ever be. That part is true. And it's entirely possible we go foom. But it's also possible we don't, and then it depends on where the asymptote lands.
0: https://www.slowboring.com/p/this-economic-myth-needs-to-go-...
That only holds if companies have a fixed need for "productivity" which is met by their current employees, such that their employees becoming more productive means they need less of them.
Every company I've ever worked for has wanted to achieve way more than they are able to get done with current resources.
But generally yes, the biggest open question about all of this is how the impact will play out on the economy, job opportunities etc. I've not seen anyone come close to a confident prediction about how this will play out.
I mean sure. Every company wants an infinite addressable market. But that doesn't mean it exists.
It might not be possible to sell 10x the software we sell today. It might not even be possible to sell 2x
Uber was basically only ever software to help people use their own cars so a very small part of their valuation was physical stuff to upkeep, it was just deals and obligations they had.
Not sure how it shakes out for Anthropic and OpenAI. There’s a lot of physical capacity that needs to be built out and can depreciate. But there’s also a lot of network effects and dependencies being built in with enterprise users.
I don’t know how swappable the tooling is either. I think over the long term the UI, model training and documentation, and infrastructure are going to end up being run by different parties and I’m not sure which leg of that chain ends up in a position to skim most of the profit off. My guess is that Apple and Google end up raking in all the money since they control the OS and app stores while the rest of the stack gets driven down to being generic commodities. At least where mass market consumer adoption is concerned.
> But then you sometimes go and talk to your senior engineering leaders and you’re saying, OK, how many projects that were on the cutting room floor got moved above the line because of the productivity gains because 25% of our code commits were via Claude Code last quarter?
> That link is not there yet, right? I think maybe implicitly there’s more that is getting shipped. But it’s very hard to draw a line between one of those stats and, OK, now we’re actually producing like 25% more useful consumer features, right? And that line is hard to draw.
That's pretty weak sauce. I don't think that justifies the headlines that came out of it, personally.
He also said in that article that what prompted the discussion was the public statement by the Uber CTO that he had already burnt through his organisations yearly AI-budget in April. Please stop this shilling mate, and trying to hide the overall perspective between this or that word.
Except that if your company go 20% faster than the others companies, you win market shares. But then, everyone will use the same tools and companies will be at even speed, but the tool will stay.
Now...if the market is saturated, it's useless to try to do things faster. Cheaper yes, but not faster.
source: https://isaiprofitable.com/
I am willing to bet a Twix we'll look back on that stuff in 2 years with a lot of embarrassment
It really does have a particular lane for each chore, and it’s reproducible.
I have a few live websites built using LLMs and they will just go for default generic templates and colours if there's no vision.
Let's put it context. Google's annual revenue seems to be north of $400B. So if OpenAI suddenly had Google's revenue, it would still be insufficient to recover their investment.
and it's a ticking time bomb because $1T in servers, CPUs, GPUs and memory is going to be worth $200B in 5 years. You can say they can keep using what they've got. Sure. But they're also not going to stop spending on new hardware. And the competitor that comes along in 5 years and spends $1T doing the exact same thing is going to have a huge advantage.
OpenAI at this point reminds me very much of the Russ Henneman pre-money hype cycle.
and in that sense, if Anthropic and OpenAI are able to create the projection that they can-be profitable despite finances seeming bubbly at best, I think that what happens is that these companies spew so much amount of content that people like Simon get into it too.
There is a deeper problem of people falling into AI psychosis too, in general, I am not sure if Simon has fallen into it or not
I think that the greatest point which can be made here is to not offload your thinking to others and to think about the situation yourself. Sounds familiar (looks like we are all off-loading our thinking itself to machines)
Side-note: As humans, we have a tendency to quickly judge or make quick decisions which stems from our times foraging and scavenging in jungles.
Another Side-note: at a certain point, I am unsure of how much to think about AI or not, certainly discussions about it that were happening 2 years ago weren't helpful in contexts that they are used now (well not in any way or form that a person discussing and getting into the weeds of AI 2 years ago is better than a person just getting into it say 2-3 months ago)
With the industry moving so fast, It is basically unsure to me of any FOMO or anything if you aren't using AI already, I find this notion naive.
People might be making strong opinions (AI psychosis) and skills on the tools available at the moment the same done 2 years ago. We don't quite know about the tech as these are still black-boxes and how they progress and what these "AI skills" might survive or not in future. Heck, we aren't even sure if these tools might survive or not or wouldn't be made magnitudes more expensive simply to break even as they are given to us for the first time at percentages of the price.
I don't know if I should form strong opinions yet and also a question of its worth so much thinking efforts in the first place, probably just gonna do my own thing (the way I want to) which includes learning C at the moment. because learning is fun.
Wait what? They spent 2 order of magnitude less on hardware.
> Gartner forecasts that large AI companies would need to earn cumulatively close to $7 trillion in AI-driven revenue through 2029, which is close to $2 trillion per year by the end of the period. In order to achieve “historic returns,” the providers would need to earn nearly $8.2 trillion in the same period.
Everyone's agency is 100% captured by belief in Wall Street. Too few <50 have any meaningful labor skills to blink.
We'll continue to have consent manufactured via media platforms and in 3 years no one will bat an eye at these companies being worth $12 trillion as Altman and Musk climb two ladders holding a "mission accomplished" banner.
I'm not even sure that 1 in 8 people I know would qualify as a knowledge worker, let alone a knowledge worker that might profoundly benefit from on-the-horizon AI. And I'm in a highly skewed population.
27% of the world's workforce is in agriculture (contrast to the US where it is 1-2%). 15% in manufacturing.
A lot of people work in "services" (especially in high income nations, where it's roughly three quarters) and some of those are knowledge workers... but a huge number of them are nail technicians or hairdressers or bartenders (etc etc).
Basically if you're not doing manual labor, it's probably knowledge work.
Roughly 1/3rd of the working population.
Some data tucked in here: https://gist.github.com/danielmiessler/2dc039762a202b083753b...
How do you know this? Im certainly open to recalibrating my numbers which is why I asked for the source
[1]: Berg, Janine and Gmyrek, Pawel, Automation Hits the Knowledge Worker: ChatGPT and the Future of Work (April 21, 2023). UN Multi-Stakeholder Forum on Science, Technology and Innovation for the SDGs (STI Forum) 2023, Available at SSRN: https://ssrn.com/abstract=4458221
https://www.gartner.com/en/newsroom/press-releases/09-24-201...
> "...with more than four-fifths of that growth coming from the emerging world."
If anyone thinks this is a part of the global TAM that's got $1000 a month to blow, well then I've got a stable of flying unicorns to sell you.
To simplify break that 1B up into 3 levels of purchasing:
1) High-tier (US, Western EU, ANZ, Japan, South Korea, Singapore, UAE, etc) - 200-250M knowledge workers.
2) Mid-tier (Eastern EU, Latin America, urban China, India tech sector, etc) - 300-400M
3) Low-tier (Rest of the world) - 300-400M
Low-tier users are mostly free tier or heavily subsidized pricing.
Mid-tier are going to account for USD sub-$100 tiers. Probably averaging less than $50/seat.
High-tier are who you are assuming is the 1B. Users are not equal in that knowledge worker count, so there aren't 1B knowledge workers to charge money.
And when you consider Low-tier users a majority of those are free users which need to be subsidized by the High-tier users. So either free tiers get much more restrictive or the providers lose additional training data. A bulk of Low-tier users cost money and provide little to no revenue.
Edit: And think about Mid-tier and Low-tier for 5 seconds. Why would they pay Anthropic or OAI when they get get 100x+ inference from DeepSeek or Xiaomi? Mid-tier may be the only area that is willing to spend money on a US provider, but I would wager significantly on the fact that users in the Low-tier almost universally do not care.
Simple - you make them work 2x, 5x, or 10x more hours.
Of course it will. The value of an employee is a multiple of what they get paid.
If you pay an employee $500k and they make $2M for your company (like Meta), then of course a 20% increase for the salary is justified if the velocity is increased 20% as well.
Imagine an employer with 10 employees paying $500k per employee and making $2M per employee in revenue (to use your numbers). They could hire two more employees and spend an extra $1M (+20%), but make an extra $4M in revenue (+20%). Alternatively, they could buy all ten employees a $100k AI subscription, for a total of $1M extra spending (+20%) but an extra $4M in revenue (+20%). You'll notice both scenarios are identical, so an employer optimizing for profit would have no reason to prefer one over the other.
The market is shrinking and saturated already and it’s not because of AI gains but geopolitical instability and supply chain issues, some of which are caused by AI spending and stupid ass PE firms refocusing on AI supply chains.
Only our pensions and futures burning.