But let's say they could someday scale that up to a much larger model, 72 large chips per wafer and each chip can do 1000 LLM requests at once (Vera Rubin?). So it's roughly the equivalent of an NVL72 rack.
You might be able to serve something like 50000-60000 requests at once. So I think it's more like handling a small city's worth of customers per wafer than the world if you had that.
I believe in less than 5 years we will get to that, but the model size and/or number of agents is going to keep going up also.
LLMs need retraining to incorporate new knowledge.
Baking them into wafers means they will be out of date by the time they finish the first wafers.
I don't see the C++ compiler standards or Newton's laws changing every day.
What I actually want is an FPGA board with a very large number of DDR3/DDR4 RAM slots arranged in banks (2, 4, 8 or even more banks). I want an FPGA board that can hold 1TB of DDR3/DDR4 RAM.
The throttling point right now is not RAM, it's bus speed. Having different busses for banks of RAM alleviates that.