The actual code and architecture is where it still lacking IMHO. Especially in rails... Like it will just build the least scalable features if you let it do it's thing. Ten queries for what could be one. No separation of concerns. Huge files, lots of duplicated code and then tens of thousands of units tests that just grow like a fatberg.
If your app does anything serious, if you have serious traffic... you are going to need to review each session finely (and your DB schema with each deploy). It could be that this is maybe an indictment of rails more than LLMs, I guess maybe time will tell.
If you don't look at your code and need to debug a production issue, do you just panic chat with your agent to solve the problem?
I guess it all depends on context, where I work ops stuff is clearly the bottleneck for a variety of reasons (technical debt that we are constantly working around, secret management for compliance reasons, etc). We self-host everything from bare metal. Some people would need to rethink the infra from the bottom up before it is "LLM ready".
That doesn't mean my job is not threatened mid/long term, in fact, thanks to LLM it is possible to rebuild that in a reasonnable amount of time I think. It's actually one of my side projects to offer this as a service. But if that doesn't work maybe I should have a plan C.
My experience is that the more structure and constraints you can place on what the LLM/Agents actually do, the better they perform. If you just let them run wild a la "make me an app that will make a million dollars MRR, no mistakes", you get a clusterfuck. And it gets more clusterfucky the longer it goes on.
This is lacking a lot of nuance. There are many types of code. There are many situations where I'm analysing something one-off and if I get 33% success ratio, but can easily verify the result, I'm happy - still saved me time and money. They're are situations where I'm generating graphs from some dataset and I don't have to trust anything - I know what the result should look like, I just need the agent to drive matplotlib. There are low stakes dashboards which I'm happy to generate and develop entirely via agents - they'll embed the updated screenshots in PRs that I can yolo-merge - worst case is that someone complains about something not working next time they visit. Then there's lots of experimenting which was never stopped to hit production anyway.
Finally after all of that you get code that's actually part of deployable features. Of course the trust is nowhere near 100%, but if you have a healthy testing process (e2e, validating different browsers, or whatever is appropriate for your environment), then what's your trust in human developer+review? Because mine is nowhere near 100% either.
In practice there are places where I extremely don't care about the code and never wanted it anyway, places where I'll read the code to check the design or just in case, and places which agents are not allowed to touch (medical billing rules for example).
Coding part of the job (as in „typing with my bare hands“) is gone. Now I get to spend more time making product decisions, capturing constraints and executable specs. Resulting code is boring, consistent and gets the job done.
That is very not true for many cases. Agents generating code are usually painfully slow, finding workflows that replace that reasoning with running code usually speed things up in my experience.
Most code just has to be good enough.
My version of "code review" is "test failure investigation" and I have a hard rule in my repos that agents never modify existing tests while they're implementing features. This means that when I run the tests after they do a bunch of stuff, I see all the tests break. Mostly they're stale assertions and we patch them up. Sometimes they're regressions and we patch those up, and sometimes I notice something dumb and dive deep into a facet of the architecture that can be improved, spend some time exploring it then get the agent to implement.
I think it's a better approach than trying to read everything and catch bugs or improve quality because you wind up focusing the things that actually matter in the real world rather than the things you think might matter.
This is how I've always approached working with offshore devs too. Focus on testing for quality control, not "code quality". After all, you're going to look at the code you wrote 5 years ago and think it's shit anyway right? So all your code is shit.
Regarding "lack of understanding", here's a recent anecdote: I had a bug in a production (but relatively new) system. The customer was texting me saying that they couldn't scan a QR code because it kept "skipping and glitching". They sent me a short video. I described the problem to the agent and it figured out WAY faster than I would have been able to that the customer's clock was set incorrectly. They turned on network time and bingo bango, the thing worked straight away.
I don't thing "comprehension debt" matters at all, because if you want to know something about the code you ask the agent. I can't remember how anything works after 12 months anyway, so I would frequently have to spend ages grepping my own code when a customer came back and asked me to change something in a system we hadn't touched since last year. Asking an agent the same thing takes minutes and is way more accurate (and fun!)
That only describes a happy path though -- I also had many instances where there's an issue and just describing it to an agent immediately identifies a fix and it all goes faster compared to me having to "load up" the flow of codebase into my brain first. But I also still have instances where it thinks it identifies an issue correctly, spits a fix which doesn't work and looks wrong. You point out that it does not make sense because of X, it agrees and spits out a new fix, which is also wrong and you start out on these back and forth wild goose chases, at this point I usually give up and do it the old fashioned way by understanding what is actually happening. If it is within an agent loop, there may be no back-and-forth to waste your time but then you pay with wasted tokens when it will eventually gives up or you stop it.
>I don't thing "comprehension debt" matters at all, because if you want to know something about the code you ask the agent. I can't remember how anything works after 12 months anyway, so I would frequently have to spend ages grepping my own code when a customer came back and asked me to change something in a system we hadn't touched since last year. Asking an agent the same thing takes minutes and is way more accurate (and fun!)
I would question the last part. In my mind the more you let go of control over your codebase the more likely that it will drift way from a place where it is still comprehensible to you, and also your comprehension skills atrophy, and with that, your ability to ask good questions and to prod your agent in a correct directions weakens, leading to more wild goose chases and burned tokens. This is all keeping in mind that for throw away or small applications, maintainability is not that of important of a value so this doesn't affect all codebases.
I don't know any more about KMP, Kotlin, Compose, Bash or React than I did a year ago. I'm not that familiar with any of those technologies (well, I am pretty familiar with bash but not to the level at which it's coded by agents), and I have never run into a situation where I felt like it would be better or faster for me to learn them and go hunting through the code to find some problem or other.
Maybe I'm heading towards a cliff somewhere in the future, but based on my experience of building software for 20+ years I'm not feeling anything but optimism so far.
:blinks: The architecture of your service relied on customer clocks being correct? That is, you built a distributed service without a clock synchronization primitive that relied on the clocks being sychronized?