Continuously updating weights will never be viable for a company like anthropic that's providing cloud inference.

So if you're hoping for that you should probably just look at the models that can be ran locally... And then you prolly realize that that's still several years into the future.

I think the end state of any productive LLM includes both: - a real-time knowledge/context graph that keeps track of state - continuously updating weights

Long term I think it's probably more interesting to explore continuously updating weights. But (and this may be yet another opinion that falls victim to the bitter lesson) even if it's _theoretically_ possible to encode everything in weights alone, I'm not sure it's not the most optimal. Today it's too expensive & finnicky to get better on your own tasks without overfitting (ie [0])

Practically (like next few months) getting the right instructions & context would be a huge improvement in capabilities.

[0] https://thinkingmachines.ai/blog/on-policy-distillation/#tra...