The context.
replyIt's impossible to train on raw document > 1MB (I don't know the exact limit, but there is one). You can't vectorize weights (or can you?!) and the output context is limited. So you can't train a big model on it's previous weight or others models weights.
The output context can be "extended" using agents but I doubt anything coherent may be outputted using that technique.
Of course this is the problem. It's amazing that for all the commentary around AI models the number of people who really understand even the basics approaches zero.
replyYeah, this was my thought (as a layman) as well.
replyIt seems some level of interpretability, even if it requires tooling, could result in the ability to spin out synthetic training data sets from other models in an automated fashion.
They might even discover some way to
program the weights proportional to an error signal
replyNow that would be crazy
I know you're being sarcastic, but I meant prompting an LLM with "generate me a X billion parameter model for doing Y task" and it would just output the weights.
replyNow THAT would be crazy