This reminds me of when "The Joel Test" was first published and it was met by people unbelieving there was a world out there who weren't even using source control at the time, let alone a bug-tracker.
Love to find some high quality youtubes to listen too to reduce my screen time.
Sure you can have linting, compilation, unit tests, e2e tests. Do you really think Anthropic doesn't do this?
In particular constructing ( and often deconstructing on change ) differential tests is very expensive relative to cheaper things to set up.
I felt this article's argument was poorly presented because of this seeming contradiction. The video I linked was much clearer about what works and what doesn't when scaling to inhuman levels of PR merges.
I also know of companies that run very expensive test suites using spot instances in AWS that cost around $100 each time. No one complains about it.
I think all the posts complaining about expenses forget to compare the orders of magnitudes.
As soon as Oracle Tests fail, the LLM has three options:
1. Rewrite the test - This undoes the benefit of the test in the first place 2. Implement workarounds and quirks to match the oracle, often at the expense of recreating buggy behaviour 3. Implement both corrected behaviour and quirks-mode behaviour to meet the oracle, raise further jobs to formalise the switch over and change of behaviour.
( And this "change of behaviour" might be a few pixels out of place! Or it could be a much more significant change. When trying to meet a goal, LLMs will often treat these with the same zeal. )
If you're working interactively, you can catch it and steer it to what's best for the feature. If you're fully automating development, then you're at the mercy of whatever the model does at that moment.
Oracle-based testing has its place, but I don't think it's good for large scale automated testing.
Also, you're talking about $500/release 2-3 times a week. That's fine, that's less than the cost of an employee.
Anthropic and others are talking about how to scale to merge 2000-3000 PRs per week.
At that scale, that cost per merge is $6M/month, surely a cost you'd want to minimise if you can cut them by half with better type-checking, static analysis, and other cheap deterministic tests, even if that's still a small cost of the whole project.
If nothing else, it's pure waste. Our planet is (mostly metaphorically) burning, and I believe that more efficient IT processes and development practices would go a long way to helping mitigate some of that.
Most jobs I worked in wouldn't bother discussing 6k of expenses. The meetings and decisions making involved probably cost more
That seems like something to pay attention to, even if you prefer to treat it as a cautionary tale with your popcorn at the ready for when you expect it all comes crashing down.
Now, are those PRs doing useless work? Are they introducing unnecessary risks or trade offs? Are they building towards something impossible to understand or that will fall apart under its own sloppy weight?
Those seem more like the right questions to me, in the long run, and I suspect that’s what you really have in mind. Because if the answers are no, then why wouldn’t you be doing as many PRs as possible?
So to me "not micromanaging PRs" doesn't make any sense at all. Bragging about how many PRs you can get merged is like bragging about how many WPM you can get when prompting an AI instead of manually typing.
It doesn't take 5 minutes to prompt claude, "Please pick up Ticket #1234567".
With modern models, you rarely need much more than that. Perhaps checking in an hour later with a quick prompt, "Progress report?" if it's still working on it.
Claude having written the ticket in the first place, defined acceptance criteria. Perhaps a human has skimmed it for a go/no-go decision, perhaps not.
The point is that at that scale you can't humanly read the code, so you need to define verification strategies that work for the machine.
Precisely so that you're not just running on pure vibes.
I can't see how I would be able to keep customers happy without actually reading the code and understand what it is doing.
Why is that scale needed in the first place?
With my skeptic hat on, it'll be interesting to see what happens when you look at their internal speed, in PRs/month vs the outward velocity in features and improvements actually shipped.
With my booster hat, it's something to learn from so I can better scale from juggling hand-reviewed claude sessions to a future it might be demanded I join at some point.
We're in a weird space in development, where the initial mechanical work that used to take a week now takes an hour at most, but what remains is a lot of manual verification and developer-testing.
If we can learn from what these mega-swarms are doing to reduce the need for manual developer testing ( because at that speed it's clear there must be close to none ), then that is a useful lesson that might reduce our work per task too and let us do more.
And yes, some of that may be stuff we've known for years, such as functional CI/CD, better linting, better automated e2e testing with playwright, etc.
But actually convincing management to invest in that kind of thing isn't always easy, especially if it risks instability of a process that from their perspective has been working fine for decades, even if that "process" is someone logging onto a server to manually fire off a script that was last touched 15 years ago.
So having talks such as this to point to is useful. You can show management an expert on a stage, and in some companies an expert on a stage can be weirdly convincing to management in a way that their own employees sometimes are not.
I don't think the low speed of development was an actual bottleneck. Coding in the real world is actually a mix of mechanical typing and information gathering. The typing part is not that relevant as it's actually not much time to type in 500 lines of code (if you touch type) assuming that's the median size of PR. It is the gathering of information that is IMO crucial for the health of the software.
It's like robots in a chemical lab. You can probably build some mechanical apparatus that can mix whatever components you want. But you still need a scientist to design experiments and to check the result. And while you can help speed up some of the process, what's valuable is the knowledge that the scientist extract, not the amount of experiments being done.
This scale of work is only possible because there's no information gathering going on. Instead it's just busy work with wishes and prayers hoping to get a good result.