I’m not sure that’s true. We have literal billions of years of training.
replyHuman "pre-training" is vastly underappreciated.
replyHell, animals are an even better example. Many animals pop out of the womb and start walking and eating and acting just like an adult!
One could argue that humans are more adaptable to novel environments
replyIf you wanna get philosophical then the decades of research that humans did is also AI "pre-training". And all of that is built on knowledge that humans have spent our whole existence amassing so maybe all of human "pre-training" is also AI pre-training.
replyA fun but silly exercise.
I'm sure we don't literally do backpropagation, but differential equations that settle toward stationary states show up a lot in biology
replyYou need to factor in the cost of training all those PhDs that never end up producing much of interest though you don't get to pick the best afterwards and claim all it took was to train him/her. Plus, once trained, it's productive only for 6 subjective hours per day (including weekends and holidays). Might have costed gazillions to train a frontier LLM but it works for millions of hours/day.
replyPHD inference doesn't scale, though.
replyI think it's fun back-of-the-napkin sometimes to compare meat to matmuls but ultimately fallacious to its core, making it an intellectual tarpit.
replyIt's much harder to argue with the math and empirical results.
Energy cost may be a pointless comparison since we don't eat electricity, but it's very reasonable to look at sample efficiency and ask what we have there that our theories miss.
reply