I’ve built a number of team-specific tools with LLM agents over the past year that save each of us tens of hours a month.
They don’t scale beyond me and my six coworkers, and were never designed to, but they solve challenges we’d previously worked through manually and allow us to focus on more important tasks.
The code may be non-optimal and won’t become the base of a new startup. I’m fine with that.
It’s also worth noting that your evidence list (increased CVEs, outages, degraded quality) is exclusively about what happens when LLMs are dropped into existing development workflows. That’s a real concern, but it’s a different conversation from whether LLMs create useful software.
My tools weren’t degraded versions of something an engineer would have built better. They’re net-new capability that was never going to get engineering resources in the first place. The counterfactual in my case isn’t “worse software”—it’s “no software.“
User count? Domain? Scope of development?
You have something in mind, obviously.
Anything that proves that LLMs increase software quality. Any software built with an LLM that is actually in production, survives maintenance, doesn't have 100 CVEs, that people actually use.
> The difference in pace of shipping with and without AI assistance is staggering.
Lets back up these statements with some evidence, something quantitative, not just what pre-IPO AI marketing blog posts are telling you.
And there are studies on this subject, like the MITRE study that shows that development speed decreases with LLMs while developers think it increases speed.
It shows, increased outages, increased vulnerabilities, windows failing to boot, windows task bar is still react native and barely works. And I have spoken to engineers at FANG companies, they are forced to use LLMs, managers are literally tracking metrics. So where is all this amazing new software and software quality or increased productivity from them?