upvote
Which is of course why, if you want to render 3d scenes to play a video game, you have to rent time on a mainframe system. I don’t see that changing ever - it’s just economies of scale!

(sarcasm, btw)

reply
The economies of scale gains are lost because you still have a middle man hosting provider who wants to profit too.

Over the long term it's always been better to buy than to rent, even if the renting option is technically more efficient on the GPUs, you don't have to pay some hosting providers profit margin.

reply
If the hosting provider can fit 1000 users onto 100 GPUs, that's enough for quite nice margins and being far cheaper than buying your own GPU.

And for users that aren't running multiple agents 24/7, you should be able to fit a good user:GPU ratio.

reply
Maybe. The economics work out better than for game streaming. When I looked in to game streaming it ended up being cheaper to buy over the long term. Though games tend to use 100% of the hardware for hours, and they tend to all be used at the same hours of the day and have to be hyper local for latency reasons. Something LLMs don’t have issues with.
reply
Things can get both more expensive and cheaper at scale, hence the term.

For example (and relevant to AI) I can generate electricity on my roof at $0.20-25/kWh, batteries included. In California the electric utility can’t offer it cheaper than $0.30-0.50/kWh. Therefore at scale, electricity is actually more expensive.

There are many such examples.

reply
Apples and Oranges. The utility uses a weird conflated fee that combines the price of the electricity and the price of connecting your house to the grid. If they split it up your marginal price per kWh would be much less.
reply
Yeah, I think the fallacy here is the conflation of scale and centralization.

Right now, there is way more scale in centralized AI than there is at the edge. But that could flip. I'd still probably put the probability that it will under 50%. But I'd also put it above zero!

reply
Setting aside that very little about economics rises to the level of "facts of nature" like physics...

What makes you so certain that economies of scale won't work the opposite way you imagine? E.g., if model improvement tapers off, but RAM costs decline (hard to believe atm, but historically likely), then eventually everyone will be able to run SOTA models on their personal hardware.

Heck, even if model sizes simply grow more slowly than RAM costs decrease, the same would happen.

reply
... said the IBM executive to a young Bill Gates.
reply