- Find a common category of error (too many tokens).
- Calculate aggregated difference between states in examples in that error category and other examples.
- Update weights to push states along that difference direction toward the non-error states.
In other words, deprived of gradient descent as a tool, it approximates it. It's clever, and it's a neat experiment. It doesn't show any superhuman insight into its weights -- with the same constraints, a human would likely take a similar approach. But it's early days for self modification and it will be interesting to see if models get a deeper understanding of weights as they get smarter.
Where are you getting this conclusion from that paper? It's both able to deduce what's happening from looking at the weights and write new weights from scratch just from a specification.
That's all we've got. I think it's a bit premature to say that it can only make improvements when there's an obvious problem.
I've been playing with very small models with Astra and it does look like the model can see things in the weights that I cannot see into.
The "fix it" loop is no longer RL in the traditional gradient descent with backprop, it can go and make targeted fixes to the weights directly instead of rerunning with new training data.
The weight collapsing loop which resulted in weight patches on top of a "super tiny model" - https://demos.isotopes.ai/pages/weighing-machine/
This makes for a "Time travel" for models easily, where I can do hotfixes on trained models & do a git bisect like test to fix issues.
2. Could lead to interpretability gains
3. Might generalize
Probably more reasons I haven't thought of. I am not an expert, so take this as layman's opinion.
Edit: Catastrophic forgetting might also be able to be mitigated.
Since the paper is also wholly and excruciatingly LLM-generated prose, it is unclear whether the author is trying to pull a fast one on his audience, or the author’s chatbot pulled a fast one on him and he is unwittingly relaying this to his audience.
0: Transcribed from the paper:
The same task and tools for
Qwen3.5 2B
Qwen3.5 4B
Qwen3.5 9B
Qwen3.5 27B
1: Transcribed from the paper: (c) The same problem, the same seed
“... If the base were changed to four you would count as follows: 1,2,3,10,11,12,13,20,... The twentieth number would be:”
Base model: no answer, cut at 16,384 tokens
“... I will write the response.
I will write the response.
I will write the response.”
After its own edit: 110 (correct) 5,341 tokens
“... The 20th number in the sequence is 110.”They seem built from traditional software dev first principles not model aided development
I am sure someone here will point out how rhetorically they are different but most of the time that's just pointing out the different words
Functionally in my machine can I use this method to extract small contexts organized similar to RAG and MCP make available and then improve local model work?
Hmm ...will have to noodle around over Thanksgiving break.
Would not be shocked too if frontier labs landed on this 6-9 months ago give headlines like Dario saying we are beyond some vague rubicon he felt mattered, then news of unprompted hacks and labs stopping models that were self improving hit the interwebz
Open models seems to be just about that far behind suggesting we're just realizing some fundamental attribute frontier labs already happened upon
My open question to any experts in this thread is whether this is intended to be a precursor to more complex self-modifications.
It’s not clear how you’d extrapolate this to more complex self-modifications or if such a thing were even possible. Think of it in terms of latent space - at some point the model has come up with as optimal a representation of the “truth” of the universe as it’s ever going to get with its current weights and structure (ie global optimum). Tweaking weights from there only decreases performance. If you did this job right you found this optimum during training.
True self-modification that’s interesting would need to be able to incrementally update the structure or alter the dimensionality.
Now the astute student will say “but there’s probably no such thing as a global optimum for every problem and you can probably tradeoff performance” but then you’d have to show this is more effective than fine tuning which does a similar job and is much more principled in how it works and more energy efficient.
To answer your question, no, I certainly do not nor do I know any other ML researcher who views this as a serious avenue at this point for any form of RSI or the likes. It is not adding new knowledge which is the massive problem at this stage to 'scaling' this approach