Aren't AI companies already butting against the limits of usable training data?
Outside of undiscovered techniques, additional costly pre-training will likely yield diminishing returns if you've already scraped the whole internet and destructively scanned as many 20th century books as you could lay your hands on.
You cannot point at Xiaomis tiny model and compare it to training a 10T param model. The levels of dunning krueger are getting out of hand.
You also arent calculating the salaries of the people involved, of which there's only a few thousand worldwide in your hiring pool.
There were probably a few dozen 20 years ago, and the existing pool grew from somewhere. AI (nee machine learning) is not rocket science - but it is a science, so it rewards those who are literate, curious, can hypothesize, experiment and learn from results to gain an intuition. I bet most statistics-adjacent professionals (millions, worldwide) can move laterally into AI research and excel at it. Nevermind AI has been among the most-in-demand undergrad programs since ChatGPT dropped. The only limited pool is for celebrity AI talent.
Your average Stats/machine learning expertise has little to do with standing up training infra and pipelines across $50 million in gpus.
stable diffusion is an excellent example, the price to train today's or yesterday's frontier is the analogy, many more people will be able to train up what we have today, especially as more of the pipeline is open sourced and the knowhow distills
Dont use llms to draft your responses to people, extermely cringe. That explains the llm induced Dunning Krueger.
What examples... There are no public numbers regarding how much it costs to train a 10T model, atleast nothing concrete.
You should just use common sense and think about how you have to fit tensors in memory to train, and the larger the model is the more GPUs you need. Then you could also use common sense to think about how GPUs have gone up in price, not down.
You are throwing around unfounded and incorrect accusations. Why? It's a weak ass rebuttal mate
common sense is using things like history to guide expectations instead just saying things without evidence that are contrary to the patterns in computing history