Boring: The documentation should be in code, not in external Word documents uploaded to the company SharePoint server.
Exciting: The documentation for the AI should be in code, not in external Word documents uploaded to the company SharePoint server.
Boring: You should give your developers the big picture of the project, not just micromanage them using Jira tasks.
Exciting: You should give your AI the big picture of the project in CLAUDE.md, not just micromanage it using prompts.
Boring: Refactoring makes your developers more productive in long term.
Exciting: Refactoring makes your AI more productive in long term.
In other words, this is not some new AI-bandwagon jumping blogger trying to rebrand old practices as something new ... this is the guy who coined the term "refactoring" arguing that best practices from 20+ years ago remain relevant.
Next up is demonstrating the AI is more productive with better programming languages.
Key parts of the abstract:
> This raises a question the programming-language community has not previously had reason to ask: should error-message detail be calibrated differently for AI agents than for humans?
> We investigate this question through a controlled experiment using Shplait, an ML-style statically typed language. We construct a suite of programs containing a single deliberate type error each, and measure how often an AI agent repairs them under ablation: a detailed error context using the unification stack; a proximate error location; a minimal type error; and a dynamic (test suite) error only. An automated oracle uses a test suite to classify each repair attempt as a type error, semantically incorrect, or semantically correct.
> We find concrete evidence that more detailed error messages generally improve an agent's ability to fix type errors. We also find that the presence of a type system appears to help more than only test suite failure reports.
That actually would be pretty exciting!
But, wouldn't AI be biased toward more popular languages, since those will by nature of their popularity provide more sources for training material?
Even still, the AI could demonstrate which algorithms or maybe patterns and techniques are more productive, in the context of the popular languages.
As an example, 110v American outlet plugs are almost certainly not the most efficient way to power devices and appliances; but, because they are so standard and are good enough, using them massively eases the burden of making, buying and selling powered appliances in the US. Similar story with USB C.
On the other hand, having a bunch of different languages is an advantage. People can pick the best tool for the job, and experiment with new language features.
It is rather different but another piece of research I liked for the same reason was this report that interviewed Boko Haram members about how they used AI to assist terrorism. You get these interminable online debates that are so unproductive and reporting that is specific is such a breath of fresh air.
An agentic refactoring pass does make sense cause one LLM reviewing work can spot things the ‘generator’ LLM missed while focused on the initial task output
But can the reviewer agent ever actually have in mind what this project actually is? And how the code all comes together to do the work involved? In other words what parts of the code are redundant or can be made more elegant
Asking coding agents to refactor your codebase is maybe like asking trauma surgeons to increase your exercise capacity. The agents are gonna need a really holistic POV to do this properly
I guess part of my point is that just splitting big files into multiple files is only refactoring in a superficial sense without having a theory of what code belongs together and what can be extracted into utility functions etc. Is splitting files actually like decomposing factors or is it like splitting a larger number into smaller numbers that still eventually get added together
A good example of what I mean is that agents often don’t ~actually~ understand the whole system anyway. They might implement a system to store and calculate something that is already being fetched via API. Humans often have a dual perspective — a holistic sense of the project and (when applying our mind to a task) a precise scalpel: ‘oh if we just look at this this JSON it has a key with this data already’
If you ask it to implement a specific kind of refactoring over a specific section of code it seems that current LLMs can do just fine. Even things as complex as "use `functools.partial` to implement the Command pattern here, rather than dataclasses".
> Is splitting files actually like decomposing factors or is it like splitting a larger number into smaller numbers that still eventually get added together
The file boundaries represent logical subsystem boundaries in the code, making it easier to reason about. The training data abundantly represents the idea of treating the contents of another file as opaque by default while other functionality in the same file can be used freely. I think it's reasonable to assume that there's something objective about the benefits that humans get from this, and it isn't just a consequence of how human cognition works.
It was helped by having refactoring experience and approaches to codebases by others, and in my case, being the original architect and being able to speak to the original and current intents, where needed.
This was using a less common, but capable and easy language for the LLM without a ton of dependancy brittleness to manage.. once the effort to remove javascript/python bias was in place, it became so powerful that once the lightbulb went on, it really got cruising.
The project was playing in the world of JSR-223 languages, where you could script in many popular languages, but it all got to run in the JVM, which was an environmental requirement.
https://en.wikipedia.org/wiki/Scripting_for_the_Java_Platfor...
It's very important that you are using coding agents with the latest frontier models and know exactly what it can't and can't do if you want to be hired in this market.
I am currently working 3 remote jobs thanks to agentic coding and one of the jobs require me to interview and hire other devs. I find it amusing how many are irrationaly resistant and unwilling to use agentic coding. They are also the loudest in the room with opinions that are outdated. On the flip side, it's just as funny how to see how complicated engineers try to make agentic coding. It doesn't have to be and that's not its design. Don't get in its way as much as possible. The frontier models are good enough that adding complexity only wastes more tokens. We've come a long way from the summer you were fiddling with Sonnet 4.5 in Cursor or windsurf.
The job market isn't suddenly going to swoop in and save you by banning AI agents. Keep up or get left behind. Don't let other people's opinions about AI agents hinder your own progress, chances are they have no skin in the game.
"enhance"
EDIT: agree with your comment updates. I'd just say it more positively leaning especially to people who are stressed financially. SOTA has advanced a lot so counter-intuitively being behind isn't really behind, it can almost be a way to leap-frog, lean on the Agents. But definitely you have to commit to putting in the reps and learning and not just mindlessly instructing AI to "fix it". Think that's where the discussion splits.
The deferral of all agency as a product owner over to the stochastic-word-box-in-a-do-loop feels like the kind of thing that, if you mentioned you do that in an interview, would be grounds to not continue.
Re: your updated comment, I'm glad you've found success with this pattern of behavior, but I'd suggest that you might be extrapolating "I haven't been punished yet for this shortcut" to "This is the future of work and those who disagree are luddites".
Refactoring towards good abstractions is more powerful than people realize. There's information theoretic bayesian math to back this up.
It's a bit of a divine coincidence that software that is more economically and energy efficient to process and run tends to also be more correct.
It's all about reducing the entropy of your code. https://benoitessiambre.com/entropy.html
In all corners of our world and the universe at large, reducing entropy in anything can be thought of as “building.”
It matches my experience which is that LLMs greatly benefit from well factored code, but are not particularly adept at creating such code.
Much like most human developers I suppose!
I still get a big win from AI on the net, but you do need to budget some time to clean up. I'm still on team "read every line".
In fact this is a case where I deliberately deferred some review because I was a blocker for another team. Now that I've got something to them I'm going back and I'm going to eat a bigger chunk of debt than I normally would, but it's worth it for unblocking the other team sooner. AI has made tech debt easier to take out, in all senses of that term.
It is also pretty decent, in my experience, at being guided into how to fix tech debt. Some other people's experience varies: https://news.ycombinator.com/item?id=49035455 YMMV.
I've had some luck while refactoring by helping it shape how to refactor, which can improve how well factored code should look like.
Providing examples of well factored code can go a long way, even if it's an open source repo of what to do / not to do.
Feature, bug or emergent property?
I don’t think the distinction matters as much as the reality.
We’re being locked into using the AI tooling bc the code was generated with AI tooling.
These giant files of doom were being generated by humans anyway and were very hard to work with. With LLMs it’s at last manageable or feasible to edit, refactor etc.
I honestly think LLMs are going to save us from ourselves as the codebases became too large and “messy” for humans to comprehend. (Mono repos of doom)
On a personal level these giant files are abhorrent but that’s just personal taste and I don’t think any of the Martin Fowler refactor/cleanup stuff is going to matter at all anymore. Kinda sad on some level.
This just fundamentally isn't true and if this is your perspective then you're using LLMs wrong.
Right, I've never seen an agent produce code anywhere near as bad as some of the human-generated code that I've worked on.
And, if your agent is producing huge files or functions, you can just tell it not to and it'll comply.
I appreciate the effort to quantify the benefit rather than pontificate. It's worth mentioning Martin Fowler wrote a whole book on refactoring [1], in which he states, "to refactor, the essential precondition is [...] solid tests", which I think is the real benefit here, AI or not. Good tests protect against regressions, whether human or robot. They also help encode the spec, which humans and robots can read.
[1] https://www.oreilly.com/library/view/refactoring-improving-t...
async fn watch_item(&self, item_id: &str, user_id: &str) -> Result<()> async fn unwatch_item(&self, item_id: &str, user_id: &str) -> Result<()> async fn watched_items_for_user(&self, user_id: &str) -> Result<Vec<String>>
This shows the limitations of vibe coding. It takes someone with a long history of software development to prompt for something like this.
Even though the model is writing 100% of the code, still needed someone with a lot of programming knowledge to write the prompt.
Most of software development is looking at some process and then decomposing it recursively until you get to those molecules/atoms of the computing world. Coding them is trivial, and while you can gain a certain boost from the AI, after a while you no longer have to write that much code. It will turn into a balancing act where the introduction of a new concept has to be done carefully.
There’s further work to do to understand exactly what’s going on here.
- the code is essentially good but all is in one file, you split it up, lines of code stay the same
- the code is essentially good but lacks some structure, for a function that does five things directly, you extract the functionality into five functions and call them from the original function, lines of code goes very slightly up
But once the code is actually bad - code duplication, bad abstractions, inefficient language use, ... - I would generally expect the lines of code to significantly drop. What scenarios are there where the code is actually bad but refactoring does not reduce the lines of code? It is certainly possible but at moment I am having a hard time comming up with a good example.
IMX: the scenarios where people hold higher standards than what you describe :) (i.e. such that "the function does five things" is deemed "actually bad". Of course, that does depend on the refactoring not causing an unacceptable performance hit, which can happen depending on the environment.)
Taking a component and turning it from 7K lines to 3K lines and maintaining functionality obviously means there’s less complexity introduced, less to go wrong now, and less overhead to modify in the future.
Sure it can go the other way, the component needs to support something it might need, we need to adjust larger patterns, this function needs to be refactored into something more robust.
But lines of code is a pretty decent metric of success for “trimming down and cleaning up” style refactoring, to me at least. It’s not everything of course, but it’s definitely an indicator.
Total LOC is a garbage metric. Things like reducing line count in specific files or components is a big benefit, but those lines are often moved, not dropped.
Since LLMs are word generators, and have a propensity to generating words, they need to be shaped to understand simplest is best, more isn't more, and less isn't more always.
Trimming down and cleaning up could be formatting, standardization, commenting, or even some basic re-architecting that was overdue.
One of the biggest benefits of llms for refactoring I'm finding is reducing technical debt.
In some projects, the code might not have much, or enough testing, or documentation/commenting.
Code is largely for others and the future if the creators of it want to move onto other projects.
Still a nice writeup and love how these meta analysises (presumeably) done via AI can now easily capture metrics that inform your workflow.
I am just sitting here waiting for Grady Booch to write "Architecture!"
> Claude is unable to look at code, look at refactorings in general and work out which are suitable to apply: a human needs to actively guide it.
Claude is happy to produce a very large Rust file. But you need human guidance to make it smaller.
Make it right is the refactoring
Why is this something users have to distinguish? Why can't these generative AIs choose good names for things?