https://xingjianbai.com/reactive-optimizers/
replyfrom this month
> fetching the relevant weights from its own network
replyThey don't have usually access to their own weights?
I think they're saying this is all happening in latent space through pure reasoning, and "fetching" is just a metaphor for that, not a literal tool call. I do believe it's possible for models to learn the capability to "introspect" their own weights during training.
replyYes, I should have been more precise - I meant the information embedded in their weights (which every neural network retrieves to function), not their literal weights.
replySo it's like a smaller LLM can learn from a bigger one, but not vice versa.
replyBut shouldn't the model-providers be the ones doing this kind of "post training learning"?
I don't know if some of the labs already knew of this. But I think there would be a limit to how big a model you can push into the context of the editor models. Typically, with 1M context and say 5120 hidden dimensions supported at fp16 / bf16, you could potentially edit the weights of an fp4 20B param model (roughly).
reply