These new tools only exacerbate this issue.
The key is to understand that computers exist to serve human needs, and the less code / UI / hoops to solve a problem the better, and to think end-to-end. If you are able to do that, most systems built by committee have a lot of low-hanging fruit that can be improved without much sacrifice. In practice, political realities and operational concerns may come in the way, but I firmly believe this is the only way to successfully operate as an engineer, and it will only become more important in the age of AI.
I push "money and data are sacred"; if we lose either of those, someone cries. Build the platform slow and fast. Governance, scaling etc sits here.
Provide reference APIs and let the product team one shot whatever they want and need, build fast and experimental, but your platform enforces constraints.
Best of both worlds (though feels like we're back in backend vs frontend development)
this is extremely true, but it seems less and less like anyone wants to invest in a solid platform and just wants to jump to executing fast and unsafe. and then pretends that ai can definitely do both when the experience is quite a bit different.
The demand for new features come from sales and executive people who want to check more boxes on a customer's checklist.
Management/Product/Product Engineer - When you are responsible for getting the work done then you look for any and every option to speed it up.
Owner/Platform/Senior Engineer - When you are responsible for ensuring it works and can be maintained then you have to pay attention to how to build it, aka the trade offs
Just because it has the power doesn't mean it should be used.
I'm not asking for everything to be some sort of beautiful code whose concision would lead Knuth to tears. I'm just asking that you not slather dozens of existing functions with three new parameters and dozens of lines of code each that basically do nothing, to cover a use case that I accidentally vaguely alluded to, if you read my prompt a certain way, but isn't really what I was interested in.
Training for good taste in architecture actually causes premature optimization at high effort allowance, but at the same time, it is combined with an inability to even recognize / exploit that later if it should ever become actually useful. Dial the effort down, and it skips the reasonable abstraction and nothing is dry, factoring is insane.
Goldilocks for simple stuff barely works with frontier models after they are out of their comfort zone for topic or line limits.. you kind of just have to decide whether it’s worth it anyway. In practice.. constant supervision for high quality code, or such high quality system design that the code quality is less of a down side.
It's no surprise that the same people are now using the same metric against LLMs except this time the LLM isn't telling them it's a daft measurement.
I think AI has been a net positive and has been used responsibly at my work. It can be used to make existing products better and more secure. You just won't naturally hear about that. There's some challenges like emerging social mores but overall good.
It always works best when layering on top of good human code. The thing is, you start to run out of that as you lean in. Tricky.
Hell, if the Dunning–Kruger effect is to be believed, it would be bold me to assume I am capable of good human code. Maybe the average is the best I'll ever get.
I think it depends how you lean in. I generated a few thousand lines of Rust code today. It was actually pretty high quality. On my initial review, I only identified one change I wanted to make.
Most functions were pretty short and well factored. I wonder if using Rust as a target helps, because (1) the training data doesn't include all the terrible Javascript and Python etc. found on the internet, and (2) Rust and many of its libraries encourage good architecture - code factored into traits, functional pipelines, a powerful type system, etc.
LLMs trained on bad code for mediocre languages naturally write corresponding quality code.
A big part of it still boils down to prompting - what you're asking it to do and how much freedom you give them. If you let them make all the architectural decisions, you're likely to end up with issues.
Why? I keep asking for examples of LLM being used to make code more simple and elegant (which I would say is adjacent to making software better, if not more secure), and never get a reply, not a single one. I'm not claiming it doesn't exist, I know it must exist. If only via a lots of careful massaging of system prompts and harnesses, even if it takes too much effort to be worth it, I would like to see it.
Of course people can't share internal examples, but this is all over the world for many months now, where are the open source examples? Not "something new someone couldn't have done before", but someone making their precious even neater (while also adding things to it, but not primarily). Put differently, can LLM delete code, can they curate it well, can they not write weird comments nobody wants or needs, that sound like echoes of the user prompts and responses to them, but just explain tricky parts of the code, like a human would?
Also, IMO a VERY useful thing to learn about a tool or a chemical or whatever is, "what happens if I keep applying it to the same small thing over and over?" -- because that's the level of scientist I never evolved past, the kind of stuff you do as a kid -- , whereas a lot of this seems to be more about how it feels to apply it here and there until it looks good, and then move on and never look back. Which really isn't much at all, especially given we're such a short time in. I mean, how useful were mugs made of lead? Are people aware of the (in hindsight) insane things that were done with x-rays as those were invented, before the potential for harm was known? I bet they had a great time playing with that stuff, for a while.
What if in 5 years it turns out that a few years ago, LLM got into this habit of implementing certain things in a certain way because it seemed the best to "them" (the training, whatever), but there's a security issue with that -- what do you do? You can only let more agents loose, and hope they don't miss anything, because you cannot even do something like "patch PHP" (which is not my thought, Casey Muratori mentioned that in on of those podcast things).
Fully leaning into LLM doing as much of the coding and understanding as can be handed off to them seems insane to me, I don't mean this to offend though. It just does, like suggesting we go to Mars, because everything is better there, and to do that, we have to blow up Earth. Or saying to magically be able to fly from now on, we need to saw off the branch we're sitting on. Even if I thought life on Mars was the future or that we could fly, if we just thought of it the right way, this isn't a way forward IMO.
Simple and elegant generally doesn't come from the LLM. It comes from me saying no to the LLM. I've discarded many branches that didn't feel like they were going the right way. Hell, I aggressively do that with my own code also. I rarely commit the first thing that works.
> where are the open source examples?
My general impression of open source is that it is, in part, a market driven by good software engineering. There's certainly below average code, but that tends to get mocked out of existence before it grows too much. Proprietary code that doesn't have a strong forcing function used to be able to hide poor engineering principles.
I think AI generated code is a regression to the mean, and the mean can be an improvement sometimes. I inherited this project. It was not originally written with AI. A lot of hard work and good engineering went into the hardware/firmware. The IoT software connecting it to our intranet, less so. Some examples that I'm fixing/fixed: ignoring HTTPS certificates, RCE by design, in-place file updating JSON files, a hand-rolled (but ineffective) fair mutex, busy waiting for a device to appear, and worst of all, no tests. None of these are hard to avoid, and even the small AI models would have come up with a better implementation.
This doesn't seem unique to my project either. I think a lot of software is just silently getting closer to the average of all open source code upon which the models were trained. I don't think I want things like SQLite to get closer to the average.
> Fully leaning into LLM doing as much of the coding and understanding as can be handed off to them seems insane to me, I don't mean this to offend though.
I think you might be reading too much into what I said. When I say "fully lean into", it's probably less than you're thinking of. I still feel that I understand what was written and can stake my reputation/career on it.
But with suitable direction, they can definitely improve codebases - especially a typical enterprise codebase, which, if you don't like LLM code, you would hate with a passion. (I can only assume you've never seen such code otherwise your perspective would be different.)
> Fully leaning into LLM doing as much of the coding and understanding as can be handed off to them seems insane to me, I don't mean this to offend though.
I essentially agree, but there's a huge spectrum before you get to that point, and lots of ways to use it effectively that doesn't create tech debt.
No, he wants a project where AI is used as a major help, but the result tend towards good code, i.e readable and well abstracted. Even if the cost is high, just an example where you can extract the best practices of using AI coding.
I specifically wrote an "initial review".
The code in question was split into modules, the largest of which was about 500 lines. Overall, the quality was higher than a lot of unaided human code that I've reviewed.
That code is greenfield code, so doesn't fit what was being requested above. But here's an example that illustrates the point I mentioned about functional pipelines:
cmd_kw("spatial_context")
.or(cmd_kw("spatial"))
.padded_by(ws())
.then_ignore(just('(').padded_by(ws()))
.then(group.separated_by(just('|').padded_by(ws())).at_least(1))
.then_ignore(just(')').padded_by(ws()))
.map(|((), groups)| Statement::SpatialContext(groups))
.labelled("spatial_context command")
.boxed()
As a result, the modules, functions, and case handlers naturally follow the single responsibility principle. This makes it easy to review. Again, this goes to the points I made in my other comment - if you give a model a good type schema, have it use good libraries in a good language, so that it has a well-structured framework in which to operate, you're likely to get good results.
Perhaps a simpler way to describe it is the old saying, "garbage in, garbage out". If you're vibe coding, say, PHP you're much less likely to get the same quality results.
Now, token calculators spew out code at a pace no human can match, which completely removes the barrier of effort - which was one of the largest natural counters to shipping massive amounts of slop.
I'll be impressed by a coding model when it writes less code than a human, and spends more time refactoring than it does just adding new layers. Until then, this vibecoding is just madness.
2. This is exacerbated by the lack of loyalty people have towards their organizations because their orgs don’t value or reward loyalty. That’s because leadership doesnt care about the long term view because of item #1. Hence, everyone is only doing short 1 - 4 year stints in order to get meaningful raises
3. The exceptions in the private sector are businesses that not publicly traded.
4. Even then, very few people care about quality that no one else can see
it really isn't a flex ppl think it is when they're burning tokens churning agents that are working for "hours" doing work
we don't need to read or write code anymore, but we do need to write (as humans) the instructions for agents.
having ADRs, product specs, acceptance criteria, etc. that are properly maintained leads to agents working for just a few minutes and pumping out reliable code.
the more vague the work for agents, the more randomness and inefficiency there will be. that leads to bugs, constantly re-writing code, etc.
imo ofc, i'm not yet running a billion dollar company so maybe i'm wrong.
Whatever, I drove down to this set of huge industrial warehouses with trucks coming and going by the dozen and found the customer service office.
“That’ll be half an hour, I recommend waiting at the cafe.”
“Half an hour for what? Aren’t all the packages right here?”
“They are, but looking up the package ID in the system takes at least 30 minutes, maybe 33.”
“33!? Wait… don’t tell me, let me guess: it has been getting slowly and steadily worse over years? Predictably taking longer and longer, right?”
“Yeah, six months ago it was 27 minutes! How did you know?”
“Umm… isn’t this a bit of drag on your operations?”
“Absolutely! Everything takes forever, every shipment is delayed, … <long rant> …”
“I… bet. Why hasn’t anyone fixed it!? It’s a one line fix!”
“No budget!”
This was one of the biggest logistics companies in the country about to undergo a multi-billion-dollar merger.
Their core business software was missing database indexes even on primary key columns!
Madness.
This also allows companies like Workforce to make billions selling the most godawful software imaginable.
We generally call it “enterprise software,” and it’s usually abysmal because the people buying/funding it are VP-level executives who care about compliance/reporting/visibility and who never have to actually use the software.
This actually comes up on HN pretty regularly. Or used to; like many topics it’s taken a back seat to AI.
i will say, at least this article says "we merge 5x more and our revert rate has stayed the same." so thats at least a metric i can start to respect
Eveyone has a backlog. What happens when you clear it in a month? Do you suddenly have better ideas about what you want to build? Do you suddenly have better intuitive designs? Do you have more users or the same users using the product twice as much? In most cases, the answer is no.
AI gives you the speed to execute when you need to, but knowing what to build and how will still remain a bottleneck.
- are the users using the new feature (repeatedly)
- what sentiment is associated with it on customer calls (extracted via call transcripts, but ideally the relevant engineer joins the call)
This is another one of the infinite meaningless metrics that AI cash burners love to use without context
1) You can merge a "new change" to fix a merge, instead of reverting
2) It's unclear whether the substance of their individual PRs are the same. Never thought I'd want an LOC metric.
3) Really, I'm going to double down on the idea that merges could be fixing merges. Also happens to increase the merge metric.
What’s the point of doing 5x more?
I don't think it makes sense to A/B test "AI-native" vs other regimes of development unless you are a large, decentralized company like an F5 company or a Big Tech. How will a team of 25 (the size of the company if you read the thread) have enough output to even A/B test the difference?
Success for what? The article opened with: "Last week Will Larson (@Lethain ) asked how leaders are evaluating engineering managers for AI skills. My reply , a list of questions to think about, got a lot of bookmarks. More broadly, I've been thinking about how to build an AI-native engineering org for a while. Sharing some lessons from the last nine months of building at FurtherAI."
The opener is "how to build an AI-native engineering org" and the context is a company of roughly 22 engineers (from a reply.) Nothing here says "evidence of success"
> I don’t see how asking how this ties back to core business results is thinking about it at the wrong level.
In an company of 22 engineers, how would you make these measurements? Would you split your company into a velocity focused group and a quality focused group and make them compete? How? What would a customer motion look like? What would these groups share? This makes no sense. This is like asking a small company to A/B test teams that used a computer vs didn't use a computer.
A larger, more mature organization has the luxury of splitting business units into AI-native vs not-AI-native workflows and comparing COGS/business objectives against each other. I know of very large tech consulting services who did exactly this around the beginning of the cloud era to figure out which aspects of their business they wanted to place on clouds and which aspects they kept on-prem, and even then that was messy because part of the value of on-prem is the scale benefit you get by owning and operating your own DC. But no 22 person engineering team I saw A/B tested this out. Companies were founded either with a cloud-first philosophy or an on-prem philosophy.
If you weren’t making intentional decisions about cloud vs on prem… why? I worked with many small businesses that did.
This feels like a value-divorced outsourced engineering mindset. You have the power to understand if things are good ideas or working!
I'm curious if you've ever worked at a startup, tried to invest in a startup, or considered purchasing a startup's product to know what it's like? The whole point is you're taking a risk trying something new and different. If you're just doing what a much bigger, established company is doing, you're fighting against an incumbent who already has much lower COGS than you.
Taking a bet on being AI-native as a startup is taking a bet that use of AI will result in lower COGS than the incumbent. As the old saying goes, "Nobody got fired for buying IBM", and yet Amazon was able to crack into that space by doing something novel. If Amazon was just IBM, why not buy IBM? The ultimate test of AI-native business models will be if these companies become the Amazon of today, or whether we'll write about them the way we wrote about "fifth generation computing" in Japan.
How's that going for you?
What can you possibly be doing as a product that justifies you releasing 60 times in a working day?
It's certainly better than the old models of releasing on a quarterly cadence with the test team and all that, but it feels like the metric has become the goal when you are releasing that often.
So I guess releasing 60 times in a working day proves that you're part of a good team /s