Isn't specialized hardware also a big risk? GPUs are more amenable to any big changes that may happen in the next 5, 10 years of AI research. Maybe we won't even be talking about LLMs anymore. Maybe matrix multiplication won't even be the main primitive.
replyIf matrix multiplication isn't the main primitive, I think we have a lot of pain coming our way.
replyMaybe that is far fetched, but I could see them specializing for some super high dimension multiplication and meanwhile 5 years later turns out "all you need" are 3x3 matrices and suddenly 90% of your specialized hardware is now dark silicon :)
replyTheir "own" as in built by Marvell and Broadcom. Especially Trainium but also TPU4.
replyThat's exactly what giant train corporations thought. "We own all the railways, we've squashed the competition"
replyand they STILL went out of business because they over-estimated the demand for their shitty rails they built to the middle of nowhere. Same with "AI."