Yes you can have your agents do refinement and refactoring. If you don’t know what they’re doing when they are doing that, they can and do introduce massive churn. The “build it fast and fix it later” crowd seemingly has never had to actually go try to fix it later, or dealt with production incidents when people have no idea how their code works. These folks are frequently promoted fast for their “productivity” and massive promises by clueless management and move on to let others have to clean up their stuff.
Where LLMs excel is in code-level bugs (as opposed to system bugs, design bugs, architecture bugs, integration bugs, etc).
Talk with an LLM, ask them to rate both code and estimate dev experience based on that code, and you will see what they value: Code that passes all the tests is a 10/10, while a codebase designed with opaque data types, Parse-Don't-Validate for all data, isolated interfaces, a built-for-replacement-not-for-extension philosphy will get a 6/10 because an out of bounds error was found.
IOW, they are very strongly tuned to value code that has no errors which can be picked up by linters or similar, while humans work the opposite way - we very highly value code that is easy to maintain, even if they do have a few errors picked up by the linter.
But that requires actually reading the output, which I am pretty certain only a rounding error of programmers are doing at this point.
Blame the benchmarks game. They're optimizing for that and that's what those things are measuring.
Reviewing and reading everything makes this feeling, yeah.
However where I can say 100% it saves time is discovery by answering these questions:
- At what state does bug X manifests?
- Explain how integration/feature works.
- I want to integrate with system X: Audit what items/features are used and what model changes are needed on my sideI at least give the new interns a stern warning: it is easy to speed yourself up by slowing others down if you pump a lot of slop.
It took about 4 days to get a production-ready reviewed code, while it took them 2-3 months to deliver something that another team judged "impossible to review".
The PR for the prototype was closed.
It helps that I'm a domain expert here, as I have a minor degree in the domain, so I can judge better. But the discrepancy is just too high to ignore.