There’s a long section in the middle where the article describes per-change verification via skill files and “ask Claude to use the chrome-for-Claude tool” when checking if changes have introduced regressions.
But there’s already tools which do this without spending tokens tokens. Playwright will achieve equal quality but deterministic results across the same verification attempts. It is write-once-use-often, so has a marginal cost near zero.
A key pillar was introducing things like static analysis to provide cheap deterministic checks as the first goal and only then moving to other forms of verification.
This reminds me of when "The Joel Test" was first published and it was met by people unbelieving there was a world out there who weren't even using source control at the time, let alone a bug-tracker.
Love to find some high quality youtubes to listen too to reduce my screen time.
Sure you can have linting, compilation, unit tests, e2e tests. Do you really think Anthropic doesn't do this?
In particular constructing ( and often deconstructing on change ) differential tests is very expensive relative to cheaper things to set up.
I felt this article's argument was poorly presented because of this seeming contradiction. The video I linked was much clearer about what works and what doesn't when scaling to inhuman levels of PR merges.
I also know of companies that run very expensive test suites using spot instances in AWS that cost around $100 each time. No one complains about it.
I think all the posts complaining about expenses forget to compare the orders of magnitudes.
As soon as Oracle Tests fail, the LLM has three options:
1. Rewrite the test - This undoes the benefit of the test in the first place 2. Implement workarounds and quirks to match the oracle, often at the expense of recreating buggy behaviour 3. Implement both corrected behaviour and quirks-mode behaviour to meet the oracle, raise further jobs to formalise the switch over and change of behaviour.
( And this "change of behaviour" might be a few pixels out of place! Or it could be a much more significant change. When trying to meet a goal, LLMs will often treat these with the same zeal. )
If you're working interactively, you can catch it and steer it to what's best for the feature. If you're fully automating development, then you're at the mercy of whatever the model does at that moment.
Oracle-based testing has its place, but I don't think it's good for large scale automated testing.
Also, you're talking about $500/release 2-3 times a week. That's fine, that's less than the cost of an employee.
Anthropic and others are talking about how to scale to merge 2000-3000 PRs per week.
At that scale, that cost per merge is $6M/month, surely a cost you'd want to minimise if you can cut them by half with better type-checking, static analysis, and other cheap deterministic tests, even if that's still a small cost of the whole project.
If nothing else, it's pure waste. Our planet is (mostly metaphorically) burning, and I believe that more efficient IT processes and development practices would go a long way to helping mitigate some of that.
Most jobs I worked in wouldn't bother discussing 6k of expenses. The meetings and decisions making involved probably cost more
That seems like something to pay attention to, even if you prefer to treat it as a cautionary tale with your popcorn at the ready for when you expect it all comes crashing down.
Now, are those PRs doing useless work? Are they introducing unnecessary risks or trade offs? Are they building towards something impossible to understand or that will fall apart under its own sloppy weight?
Those seem more like the right questions to me, in the long run, and I suspect that’s what you really have in mind. Because if the answers are no, then why wouldn’t you be doing as many PRs as possible?
So to me "not micromanaging PRs" doesn't make any sense at all. Bragging about how many PRs you can get merged is like bragging about how many WPM you can get when prompting an AI instead of manually typing.
It doesn't take 5 minutes to prompt claude, "Please pick up Ticket #1234567".
With modern models, you rarely need much more than that. Perhaps checking in an hour later with a quick prompt, "Progress report?" if it's still working on it.
Claude having written the ticket in the first place, defined acceptance criteria. Perhaps a human has skimmed it for a go/no-go decision, perhaps not.
The point is that at that scale you can't humanly read the code, so you need to define verification strategies that work for the machine.
Precisely so that you're not just running on pure vibes.
I can't see how I would be able to keep customers happy without actually reading the code and understand what it is doing.
Why is that scale needed in the first place?
With my skeptic hat on, it'll be interesting to see what happens when you look at their internal speed, in PRs/month vs the outward velocity in features and improvements actually shipped.
With my booster hat, it's something to learn from so I can better scale from juggling hand-reviewed claude sessions to a future it might be demanded I join at some point.
We're in a weird space in development, where the initial mechanical work that used to take a week now takes an hour at most, but what remains is a lot of manual verification and developer-testing.
If we can learn from what these mega-swarms are doing to reduce the need for manual developer testing ( because at that speed it's clear there must be close to none ), then that is a useful lesson that might reduce our work per task too and let us do more.
And yes, some of that may be stuff we've known for years, such as functional CI/CD, better linting, better automated e2e testing with playwright, etc.
But actually convincing management to invest in that kind of thing isn't always easy, especially if it risks instability of a process that from their perspective has been working fine for decades, even if that "process" is someone logging onto a server to manually fire off a script that was last touched 15 years ago.
So having talks such as this to point to is useful. You can show management an expert on a stage, and in some companies an expert on a stage can be weirdly convincing to management in a way that their own employees sometimes are not.
I don't think the low speed of development was an actual bottleneck. Coding in the real world is actually a mix of mechanical typing and information gathering. The typing part is not that relevant as it's actually not much time to type in 500 lines of code (if you touch type) assuming that's the median size of PR. It is the gathering of information that is IMO crucial for the health of the software.
It's like robots in a chemical lab. You can probably build some mechanical apparatus that can mix whatever components you want. But you still need a scientist to design experiments and to check the result. And while you can help speed up some of the process, what's valuable is the knowledge that the scientist extract, not the amount of experiments being done.
This scale of work is only possible because there's no information gathering going on. Instead it's just busy work with wishes and prayers hoping to get a good result.
I think tokens are near-free already, even for those of us outside Anthropic.
More importantly, your philosophy comes from a time where thinking had to be front-loaded: you try to predict everything that can go wrong, then design deterministic tests that can catch those things and a decision tree to figure out what to do in each case. This approach was the only viable way until now, but maybe that will change. Having the equivalent of a senior engineer thoroughly review each error with full context of what the application and feature is trying to achieve could perhaps uncover different insights, or cover edge cases more smoothly.
How long do you think that will last?
Am over 11 billion tokens this month on a subscription. In API usage its close to $5000 and there's still 13 days left in the month.
Eventually the music in musical chairs stops. Already many anecdotal reports of the always opaque limits being lowered across the board. On a gemini pro account for personal use I can use a weeks worth of Argon in about 20 minutes on agy in the last month.
ChatGPT Pro I can use the weekly limit in under 2 days, this is with significant context clearing and handoffs.
The cost to provide frontier models isn't cheap and it's entirely subsidised by a winner-takes-all mentality from investors which in all likelihood won't come about.
Many benchmarks are already saturated today on many tedious tasks still being done by humans across the world. Frontier labs are introducing their own budget options prior to IPO, undercutting their revenue, because otherwise they risk losing seats.
Choosing what to do and how to do it is the actual frontier now and is still not apparently well supported in any model. Current so-called decision models are not decision models, they are available-fact-evaluating models.
No one is debating this. The debate is that an agent performing validation work via its own chrome plugin and internal reasoning will always cost more than performing that same validation work with deterministic tools.
right?
In this age of AI, anything well understood and effective while being efficient is out. These tokens must be used, atmosphere must be heated.
It's November. Winter is coming.
Climate change and global warming is really a scam, they have made it snow with chemtrails in the 80s, it seems.
Though I miss snow and cold. They are not making seasons as they used to anymore. They shouldn't have privatized weather.
(This comment is written by a human, and contains an implicit /s)
No real software engineer think that way, as things that can go wrong are near infinite. Instead you design the right thing and write tests that verify that it’s indeed the right thing and add some more that are the wrong thing, especially at the edge of definition. Then you rely on the dev to not make very obvious errors like coding something that is not in the requirements or is completely opposite.
Gong to be honest where do you think this is going?
Do you think tokens are going to magically become more expensive or models are going to become less efficient?
Of course you should be tokenmaxxing, you can already tokenmax with a model that competes with the SOTA 1 year ago on a moderately powerful laptop you can walk out of an Apple store with.
I, personally, have already experienced situations where I was employing cost-optimization optimization strategies for models 8 months ago, that are borderline-useless now because the Pareto frontier has advanced both more than I thought and also in different ways than I thought (cheap models got smarter, better at following instructions, and more clever about tool calls).
That is, the strategies decreased in effectiveness disproportionately to how much the model improved.
It's worth considering that strategies that seem insanely expensive now may be economically viable in a year or two. Or possibly significantly sooner than that.
It's also possible that, like how jet engines get more efficient as they get larger (leading to the 737 MAX incidents, which seems spiritually significant) corporations with budgets orders of magnitude greater than ours benefit from very different strategies than we do.
You're assuming a lot of costs that should just be considered ok in your stance:
* Why pay anything multiple times when you can replicate deterministically for free? * Why require always-online models when you can run computation locally? * Inversely, why use power-hungry/PU-hungry models locally when you can have a deterministic software that can be optimized to not* be memory/PU/power hungry to do validation?
I think tokenmaxxing can be a great start* to get things going but when it comes to productizing I want determinism and cheap optimizations, and using frontier models to build these cheap optimizations is what we've been doing at our company over this past year to great success.
I just told him to make the AI write a small selenium python script to do the clicking and it worked amazingly.
[ AI ]
[ Software ]
[ Kernel/OS]
[ Hardware ]For me this article is more aimed at devs that are still stuck treating code generation and verification as two separate tasks. Giving agents the right tools (including playwright) to do both at once makes things more token efficient, not less.
Where deepseek prices inference the way systems programmers price latency. As a resource to be spent carefully towards specific goals, with systems in place to ensure you're within budget and getting the value you expect from it.
And then this also has second order effects on how each company treats human attention. Having fable xhigh do 3-4 passes over a change can legitimately reduce the need for human attention on it in a way that deepseek flash just can't really. So then they price operator attention in a similar way. With ant nearly considering human intervention a failure mode, while ds again prices it as a budget to be designed within.
And neither one is "right" speaking generally. I have my aesthetic preferences and my own constraints that make one a more appealing approach personally. But both are building towards the goals they have with consideration of the resources at their disposal, it's good engineering either way.
I think future AI cost will be as cheap as doing an API call in classic sense.
So no way it will be as cheap. That is just wishfull thinking.
In the future when AI is able to do everything, there will not be one single super-AI that does it all, but a whole ecosystem of different AI. This is the same as how in the biological world, while there are various degrees of generalists, there's no single cell, plant, or animal which does absolutely everything. If our body needs more blood, it makes more blood cells, it doesn't repurpose intestine or skin for the task.
Similar argument in economics, comparative advantage: an AI which can do everything is best used for the things which only it and no other AI can do, and leaving to other (smaller, cheaper) AI relatively simple tasks like face recognition.
Even where AI can do this, that doesn't mean the best solution is AI rather than a traditional API.
First, last I checked AlphaZero is a much better chess player than any LLM, and by a large margin.
Second, this isn't even about finding the best model for small tasks, it's about best use of finite resources, compute or energy, which means that it still holds even if a future AI (call it Alice) did everything better than AlphaZero does chess, it would still make sense for Alice to offload chess to a merely-good model like AlphaZero so that Alice could spend more time doing the things that nothing else besides Alice could do:
On some level you are right, LLMs and the agents built with the various models and architectures will get cheaper. But there are some significant drawbacks by introducing this risk, regardless of cost. The future will need the capacity to understand this too.
This is the process of increasing efficiency today, automate manual processes to reduce human effort. Why wouldn’t models get better at this as well?
If you're doing this, the agents should create deterministic tests from its process rather than Agent tests or just writes deterministic tests for start. This is a bit of an absurd approach.
The issue I find with these articles is that they are basically the same boat as "Facebook Engineering" articles were 20 years ago.
People will be "oh because Anthropic does it it must be the way", then a bunch of influencers will retweet it as "anthropic kills writing tests", which some CEO's and team leads will parrot and get their team to absorb these practices even tho they are not the best practices.
Some of these things are just reinventing the wheel, like this is a basic script called "run_simulator.sh" and a pre-commit test hook, not a skill and "commit gate" that needs to check session transcript:
>To codify how to start the simulator or run your app, create a skill called verify. Put it in a .claude/skills/ folder in your repo, as close to the relevant code as you can, and add as many as you need. Claude knows to use the verify skill before committing changes.
>We've also created a commit gate that can block a commit if the session did not contain /code-review or /verify. We've set it up with off, observe, and enforce modes.
Scandalous.
Do you not see the propaganda through the pseudo helpfulness?
I find it amusing that people phrase bog-standard "what can our product do" documents as "propaganda", even when it's a dogfooding exercise.
Like, I guess? In the sense that all marketing can be seen as capitalist propaganda?
I think it's worth designing around that because tokens will only get cheaper over time.
The reason you should use Claude and not playwright always is because the the invariant you wanna capture is at the natural language level.
Like imagine I wanted to verify that Claude Code works and doesn't break. I can describe the invariant I want in natural language. Like: make sure that subagents don't block the original thread.
Doing this in playwright means that you have already bought into some previously existing architecture which means your tests are now correlated to your code. Having the invariants written in natural language allows you to decouple code vs the invariant.
I expect a lot of greybeard conservative programmers to keep opposing using AI here until eventually, not using AI will be seen as more risky.
The issue with only having the natural language specification is that it is non-deterministic.
I expect a lot of no-beard man-child programmer wannabes will brain rot early in their careers and find themselves wallowing in the hopeless despair of sitting in front of a computer arguing with a chatbot for decades on end, living a life trapped under the weight of endless mediocrity, unknowingly desperate for a modicum intellectual stimulation that you’ll never find because you can’t prompt for that, bro.
That was kind of my point. Natural language almost always survives rearchitectures because your higher level interface to your end customer rarely has churn.
In a few years, agent driven testing will become normal and almost expected in some enterprise companies. Obviously we will still have people like you opposing it on grounds of “but just write tests in playwright”.
I also find the greybeard archetype fascinating. They’ve opposed almost everything new including Cloud, microservices, kubernetes. Now with AI. And here in this thread with AI driven tests. They seem to think programming should be done exactly how it was done when they were in the prime of their career. Anything before that is archaic and anything after is hype.
Not that it is an explicit, conscious direction, but it very much is one.
You’re not wrong, in that free flowing tokens are enough to convince such a group it’s simply good advice.
The more you use product X the more product X you need to buy. Nice to be selling product X.
I'm not working for a manager, so I don't need to use it at all.
It can meaningfully “type” faster than you.
Take the handicap if you really wish to, I suppose.
I still use LLMs (cautiously) for searching and troubleshooting API things.
Same here. But as opposed to the "use AI" vs. "don't use AI" crowds, I'd prefer a more pragmatic approach:
Do whatever possible to maximize the quality of the software. And with quality I mean correctness, reliability.
AI is just another tool there. Where it helps produce more correct, reliable code, consider using it. Where it doesn't, don't.
Sadly it seems many companies have a different definition of "better" (featuritis, shortest time to market, developer productivity, etc).
I’m banking on that being an evolving, but necessary skillset. For a while at least. And I’m still gaining experience in the process.
Plenty of exposure, plenty of prior experience, it’s a weird mix of management-ish context handling and sme mentorship while also providing the research benefits you mentioned.
I stopped using as my daily driver around 9 months ago, and at that time it felt very heavy/janky and seemed to have very regular bugs and breaking UI changes.
I’d be happy to hear that it has gotten better!
I definitely wouldn't take their word on verification as a particularly good source.
most of that is prob for internal tooling and model related work, but seeing their public facing software makes me think that all this work hasn’t really translated to all that much more than 3 years ago
Im already spending my $2k usd per month in tokens with just one or two agents. How do they expect me to do that?
I have worked in companies that spend millions a month on some random infra. I'm obviously not going to complain that its not sustainable?!
Edit: can't reply but Token maxxing is not over. We are just getting started. And please keep in mind two things are happening
1. AI costs are reducing by 2x every few months
2. This causes more usage because of Jevons Paradox
3. Also, running verification tests on releases is not that frequent and it happens only once for a whole team of 5-6 people
If they spend tokens like Anthropic suggests, they won't be profitable for long.
If you want Shakespeare, you need to trend closer and closer towards infinite agents, duh.
The future of IA would actually be to large scale prototype easily, even more like Python was a prototype language.
Whether a human can or should review every diff is a conscious project level decision. I have yet to see a project where it is not feasible. The trick is to break things into small enough pieces that you can review them. Just like one would do if you were doing such changes manually. Nobody in their right mind performs non-mechanical changes of 100s of files.
To companies, some risks are worth taking, if it can increase revenue or lower costs. Many companies happily do not-quite-legal things, taking into account the potential of being caught and the size of the fine.
In the case of Anthropic, from that perspective, it almost seems like a no-brainer that they would want to push an "our AI can do it all, no humans needed" narrative in everything they do. They need hype like we need oxygen.
I would agree that for many others, the risk is too high, and the reward too low. So they'd indeed have to be out of their mind to deploy code nobody understands. But depends on the situation of course.
Just trying to say, in a really lengthy and possibly patronising fashion, that "good engineering" is unfortunately not something many (most?) companies prioritise.
We're in a brave new world, many many big name companies are doing this.
Not 6 months ago, sure. Today? It's not totally crazy. In a year it will be routine.
If the tests are hermetic, you should always be able to find the root cause (by bisecting to determine when the test became flaky).
If the tests are not hermetic, it should still be possible to identify the dependency that is making them flaky, and use that as a justification if it still makes sense to delete the test.
Perhaps we need a level above linters: a tool that constructs the abstract syntax tree and verifies its various branches against deterministic rules.
How to do this for architecture: unsure.
Isn’t that basically a linter?? If you go too far outside a regular ol’ linter (even a real nice type aware one like typescript-eslint) you hit a ceiling where some intelligent being needs to interpret things.
Though I will say I have seen interesting projects in clojure world like rewrite-clj, which makes it really easy to do structural analysis and modification of clojure programs. I’ve definitely thought there could be more room for stuff like that in the age of LLMs. I’ve also made more use out of ts-morph than ever before.
Also in my box of purely delusional ideas, I’ve thought about how cool it would be to encode LLM-derived semantics at the AST level, which then could be run through a regular deterministic linter. Like a switch case or pattern match could have “this is the happy path” or “this is an edge case” encoded into each branch. Then you could do stuff like collapse all non-happy-path lines, or highlight all edge case handling.
https://github.com/gregwebs/skills-sdlc/blob/main/skills/ver...
The implementation process that I am using also looks for verifications during planning and reviewing and makes sure they are completed. It does use a lot more time and AI spend than just asking AI to do something. I still do the latter for simple or noncritical changes. I had to switch to using DeepSeek as my workhorse to keep down my spend.
https://github.com/gregwebs/skills-sdlc/blob/main/skills/imp...
They won’t even do all this verification nonsense and their end product will be of superior quality.
Who ever had the time and will to write unit tests that mock every single function/class being called, checks the strings of thrown errors, etc., for every single function, for several tests per function?
The instant you change the implementation the tests break because it's testing how many times each method has been called of an object that was passed as argument.
That's just an insane way of writing tests. But for AI that's normal for some reason.
It feels like it's making unit tests the "textbook" way that in practice you wouldn't do because in practice half the things it's testing are inconsequential.
this u?
Did you forget that they literally publish benchmarks that show how much of the task can be done independently?
> You can also close the verification loop by using the results to improve the agent’s performance on that task. For example, if an agent repeatedly fails a test that checks logs for user data, you can create a hook or a more explicit rule in your CLAUDE.md to avoid future issues. This (mostly) stops issues from happening again.
There are people out there who are doing all sorts of crazy neat things in the verification space using the tools you build and yet the people who could be helping seem to have no good ideas on how to help :/
How about making Claude generate scripts that you can run?
Algorithms are better than agents. In fact, the whole point of AI is to reduce thinking to algorithmics. I didn't see a lot of evidence that the team understands this. If you need a magic box to test your other magic box, you have magic boxes all the way down.
Anthropic operates at pretty big scale, and the original question was about engineering background.
we don’t actually do any of the work but we think about it
"The resource at “https://claude.dev/_next/static/chunks/241gnzjfqbp9f.css” preloaded with link preload was not used within a few seconds. Make sure all attributes of the preload tag are set correctly."
I am more interested in the remaining 10%.
“Also Claude, but we're fixing up the attribution to be super confident before claiming it”
https://x.com/bcherny/status/2111207278653399460?a=CQxYPj8Va...
More snake oil and theatrical agent setups described in insufferable Claudish. I think the target group of Claude is people with an IQ below 100.