There's a law of diminishing returns at play here, and doubling the energy cost of training to wring 2% more performance out of the technology isn't going to be very useful, because most of the problems it is capable of solving will be solvable with the previous-gen 98%-as-good model.
("there's a law of diminishing returns at play here" is an article of faith. But then, so is the belief that these models will keep getting better).
lol You can always tell who has never ran a business before with comments like this
(I don't think it will work).
Are there any use cases that have enabled one customer of a frontier LLM to outperform a competitor using a different frontier LLM? Or is this why we are seeing confected points of comparison like solving challenge problems in mathematics?
There are going to still be worthwhile improvements but they are going to be more like not how to make transformers 10x cheaper but how to make next training run cost 9 trillions instead of 10 with a very particular optimization designed at the cost of hundreds of millions for this one specific run.
I could imagine a belt of data centres around the equator, that hand off their computational loads as the sun sets. Good scifi-esque premise.