upvote
The memory makers will not expand demand drastically. It is in the nature of their business to keep the market under-supplied, otherwise the following oversupply will kill them. Instead, supply is just rerouted from less profitable segments such as mobile and personal computing.
reply
China is about to flood the market and prove this notion wrong. If there is demand they want to meet it with supply.

But to your point, that is exactly how American companies like to play now. No one is stopping them from screwing over the consumer.

I have a Micron near me and they are building another chip facility but we are years away still so I suspect China will beat them to the punch.

reply
Supply and demand always balance out. There is no way manufacturers aren’t going to compete away these inflated margins, as long as they feel like this demand is sustainable.
reply
There's very few manufacturers, I believe 3 globally? And there's a large moat. Nobody can compete with them in the next 10 years. It's really not hard to coordinate action between 3 companies.
reply
Apple could always decide to build their own fab or some such thing.
reply
It sure looks like Sam Altman's masterful gambit to corner the memory market has had unforeseen consequences.
reply
What’s the lifespan/refurbishability of the capex elements like the “GPU” modules or even the DRAM soldered into them?
reply
For lifespan, AWS is still running a ton of T4 GPUs from 2018, that power a lot of computer vision models. A ton of these will have a long life, not all ML is about frontier LLMs.
reply
I wonder if we will see an adoption of alternative floating point formats. IEEE floats are notoriously terrible at lower widths (<= 16 bits). Floating point formats such as posits do much better at 16 or 8 bits. If you could train at 16 bits per value instead of 32, and suffer a much smaller inaccuracy penalty than you would from IEEE32 to IEEE16...
reply
For some reason I still haven't heard any predictions on when new fabs will come online to meet the current demand. This shouldn't be too hard to find out, since the building time of fabs is very predictable process.

The difficult question is more whether foreseeable memory demand will remain at the current level, grow even further, or shrink again.

reply
Supply will not meet demand. What incentive do the handful of dram manufacturers have to end the party? This is what happens when legal monopolies finally win control. Dont't worry. The patents will expire in a few decades. Our grandkids will see DDR5 get cheap again. The system functions as intended.
reply
Patents is not the issue here. Not even close.

The up-front investment of a memory fab is measured in billions, and takes years to construct and get running. The margin on the chips themselves is terrible, so without scale its not worth even trying. DDR5 is a industry standard that takes some effort to conform to, but the licence fees is a drop in the bucket to the cost of creating a fab.

The fabricators were cautious about increasing production, and slow to start planning. It takes further time to build up capacity, and if the demand drops down, they may end up producing dram at a loss when the market flips over to oversupply. The demand whiplash could kill any company that dared betting on increasing production. See the "bullwhip effect" https://en.wikipedia.org/wiki/Bullwhip_effect which has killed semiconductor fabricators before.

There is a discussion to be had about how to maintain national semiconductor production in Europe and US as a strategic industry, but historic attempts have all failed.

reply
I have fairly simplistic view of the economics involved here. Could you explain why the ability to sell more chips wouldn't be sufficient enough incentive to increase supply?
reply
Not the person you’re replying to, but RAM has historically been a boom-or-bust business, and companies that invest to meet demand during a boom cycle usually have that new capacity come online just in time for the bust.

If it was just variable costs and new capacity was available today they’d do it. But there are substantial fixed costs and delays to increasing capacity, and that uncertainty makes it risky.

reply
Bringing on new fabs takes many years and billions of dollars. You're exposing yourself to a lot of risk if you build now and find that the gold rush is over by the time your new capacity is online.
reply
> a path to a ~3x hardware cost reduction

Really?

How long do we have to wait until that ... cost reduction hits us?

reply
For supply to meet demand. Depends very much on how aggressively producers scale and on how demand grows or shrinks.

Safe to say at least a year or two. It'd be shocking if it took a decade.

reply
2-3x is completely dwarfed by the remaining improvements in training which is still in its infancy relatively
reply
Probably, but at some point we're very likely to run out of significant training improvements and it's not clear that we'll see that point coming from a long way out.

Likewise it's probably dwarfed by improvements in how we make dram - continuing the roughly exponential (maybe a bit less recently) scaling of chips - but not necessarily.

The 2x from returning to previous costs is interesting because it's practically guaranteed, and it's on top of everything else. We're just currently "overpaying" (relative to the stable market price) for the manufacture of dram because of a sudden increase in demand.

reply
Unless there's a new paradigm, scaling up is all they can do to improve performance. They've shrunk down all the way to 1-bit models and all the low-hanging fruit is gone. There's no way for them to get much smaller, so they have to get bigger and faster to meet expectations.
reply