Abstractions are for humans.
I disagree with this. The mistake rate goes up rapidly as context grows. Current frontier models get noticeably more stupid at context sizes of only 100k tokens or so, and inference becomes slower as the context grows, leading to more rework and more token usage.
I foresee a trend of having _smaller_ code units, more modularity, more abstraction, because agent work on smaller units leads to smaller contexts and a better chance of the agent getting the change right first time. Or a higher likelihood of a lower skill, lower cost model getting the change right.
It may be domain/task/prompting dependent etc, but my experience is different. I haven't subjectively felt any degradation up to like 500-600k tokens with current Opus/Fable models. After which I usually do a manual hand-off to a fresh session anyway. I don't use context compaction though.
The first attempts I made to one shot improvements to the recompiler before doing the refactoring and test framework were basically all wrong and introduced lots of bugs, a per turn success rate in the single digit percentages. Each of the non-functional changes I made improved the performance substantially as you might expect. The real eye opener though was the refactoring into smaller units - that bumped up the success rate much more than I was expecting.
Don't get me wrong, the success rate of human changes is abysmal on these tasks, which is why the recompiler was in such a sorry state already. There are edge cases and gotchas everywhere. Opus, Astra and Gemini 4 all reason for hours on individual changes, and lesser models can churn for much longer and still fail.
Interestingly the code itself isn't really all that big - about 45000 lines of C all told.
It's tiny - 7 items or so, if you are talking short term memory, especially when you compare it to an LLM's context window.
But humans transfer important items to long term memory, which is huge. We have over 100 trillion synapses, compared to an LLMs 10 trillion or so weights. LLMs can't reprogram their weights.
Which means after a human has worked on a system for a few years he will know far more about it than today's LLMs can ever know, because they won't be trained on bespoke systems.
But almost certainly not more then a AI trained on the system could know - and probably a very small AI at that. That was demonstrated by AlphaGoZero. It is tiny by today's standards, but knew more about playing Go than any human because they can't remember enough about it.
It's seem likely that today's cloud LLMs will, like AphaGoZero, know more about how to write good programs than any human. But they will never be able to build a big system as well as a human.
I dunno what happens tomorrow. Do we train in-house AI's to maintain our in-house computer systems, run the company, or even the country? Maybe that is the end game. We would probably start with the country, because it's easier to justify the huge expense. But I suspect if we do, they will be different from today's LLMs. You will be able to train them incrementally, from a single example. Maybe they will call the training dreaming.