Step 3: Prompt the agent to write mutation tests to harden the suite
Step 4: Prompt the agent to write property-based tests.
Step 5: Prompt the agent for 100% code coverage
Step 6: Keep a look out for small issues with steps 2-5 above and refine a Skill or Rule to package the workflow and plug gaps in the model's current shortcomings.
Step 7: Dedicate some time to manually reviewing and thinking about interesting edge cases worth testing.
---
If you follow these steps, you will have better test coverage than 99.9% of any code an engineer ever wrote pre-AI... and it will be almost completely automated.
My team's incidents and bugs are DOWN this year while we're shipping more.
Leveraging AI effectively requires a new paradigm of interaction. The name of the game is to use techniques and strategies and guardrails that create an environment that produce good software.
I don’t think anyone has figured out the proper solution yet, I personally try to get the LLM to write code similar as I would have written it myself.
If you mean asserting against hardcoded literals, then that is the preferred testing shape in most cases. And for good reason.
If you change production code and a unit test fails, that is useful. Idk why that's silly. If you want to test more complex interaction write a UITest or integration test.
I honestly think we're all living in different worlds. There's probably large differences in capability on a per-language, per-team, per-codebase, per-engineer environment that are producing wildly different perceptions in output quality.
And yes everyone works in different domains. I work at a database vendor myself, so mostly lower-level work.
I’m very happy and successful with LLMs, I just don’t think that these “just move all your tests to your type system!” or “get 100% test coverage” rules are an answer in the long term as we haven’t yet figured out how to do this well.
I personally treat tests as components that need to be well architected, maintained, and adding them risks increasing technical debt and needs to be treated as such.
I was describing general strategy, that when applied in concert is more than the sum of its parts.
E.g I mentioned property-based testing, that is another angle at which a test avoids assertions that are too fine grained.
I think it has much more to do with overall architecture and design, and as for now, human oversight.
just curious, and tempted to do the same, so far I don't like the code made by llms in dynamic languages codebases
Another of the shallow arguments thrown around, similar to "we don't check compiler output so..."
People learn and improve. LLMs don't learn, they follow instructions to a degree.
A human has to wrestle with it to work how you want, ergo humans are necessary already.
You can't fire the LLM and hire a competent one. The few available is all there is, they're all copies of the same thing.
You don't have a team. If you think 10 LLMs = a team of 10 people, then you don't know what a team is, how different skills and views make a big difference to outcomes.
LLMs can't innovate. They can't come to you with a great idea they came up with. They don't have ideas.
If none of these things are worth anything in your particular case, then that's ok. But let's not actually lose our minds and forget why people are intrinsically better than machines.
There are only a few arguments to support using LLMs, all of which are from people who a) want to spend less on salaries or b) sell you LLMs.
Teams are often hierarchical where the leaders define everything and people under them have less things to say about it. Not much different than an LLM team.
I do a lot of LLM coding and I catch about one of these per week. But I still look at the code after the LLM writes and reviews it. So I'm going slower than most on here, so my default assumption would be theirs are doing this more often that once/week. Whether or not it matters depends upon your project.
My favorite one from the last month was the LLM designed monster hit points in a way that everyone is healed to full between combat. No, it was not intentional, HP were just stored in a way that the structure storing it completely disappeared outside actual combat.
The belief that AI can do all that without the gates is throwing away something useful where people get to understand how the system is changing. If you keep the gates but use AI to generate all the code you either slow AI down to the speed of humans or you end up with a massive backlog of stuff to review.
This is a hard problem.
They aren't going to be creating 100 liabilities/hour?[1]
--------------------
[1] Each line of code added is a liability. We appear to have forgotten that.
Not saying it's better or worse. But it's definitely different.
My first startup was building requirements management software for small businesses to solve that exact problem. Devs add things in all the time. It's usually referred to as 'gold plating' or scope creep.
In general any type system is designed to prove things. So all I'm really saying is "use strong types to prove more" which is irrefutable.
Type systems are not a panacea. You still have to be logical and careful about them, especially around boundaries where types are just assumptions and not logical inferences. Then you have functions that switch type, but only a subset of the input is actually valid (string -> int for a number parser). Checking for NaN is like checking for Null.
Type systems does help, but they need the same attention as code to be actually helpful.