undefined

points

[-]

And test creation and maintenance. AI-generated tests will often embed existing bugs as testable patterns that fail when fixed or rely on empty asserts to increase coverage. A human author generally knows better. Coverage is increasingly becoming theater and you have runtime testing eating up CI resources and time with bloated, unnecessary, untargeted testslop that haven't been mutation tested or a human eye assessing if the business logic is actually represented.

It's really the smaller firms which are able to learn quick adaptations that will win out in the long-run of this thing. Legacy firms with institutionalized bureaucracies are relying on gobbling up competitors and creating vertical monopolies while pushing out the worst slop - it's why GH goes down, why Meta products are buggy as hell now, and everything feels like it's unpolished MVPs these days.

by bunderbunder2 hours ago|

parent|

[-]

So far this year, of the defects severe enough to warrant a formal RCA process that I’ve dealt with, “AI generated tests asserted that the defective behavior was correct,” was a contributing factor in every single one.

by desolate_muffin29 minutes ago|

parent|

[-]

I haven’t had this happen yet, but I was just thinking about this last night. I would be very interested to see if exercising TDD, as difficult as that practice is to actually do, would have a meaningful positive impact on improving this phenomenon. Usually agents model what most humans do, writing tests to pass and assert that the code you just wrote works as you wrote it.

by bunderbunder3 minutes ago|

parent|

[-]

I’ve had decent luck with it as long as I at least stub out the test cases with a test name and a “given … when … then” comment.

Otherwise it seems the AI has a tendency to write the worst kind of TDD antipattern microtests that simultaneously make refactoring impossible while failing to document or even confirm the most important functional requirements.

Once it’s got those stubs it does do a pretty good job of suggesting behaviors I forgot to specify, and other improvements. It’s just “designing” test fixtures from the ground up that seems to put it into “generate plausible-looking slop” mode.

by btown3 hours ago|

parent|

prev|

[-]

This. The ability to know that a vibe coded PR is all right to merge, is a function of both the thoughtfulness and simplicity of the underlying architecture, and the level of ownership that contributors have of the underlying products. Both benefit more nimble teams.

by cykros59 minutes ago|

prev|

[-]

Agentic code review can also be employed, but it's worth noting that, as the so-called "Bitcoin Red Team" (which has been testing all of Bitcoin related software in the wake of the July 30 Coldcard attack) has discovered a very useful thing to consider: American models are absolute trash when it comes to reviewing code with an eye on security, due to their "safeguards." This applies even when the code being scanned for vulnerabilities was just written by the same agent.

Kimi K3 seems to be the favorite there, though I believe some of the other open weight/open source models are being employed a bit as well.

And yes, even that requires human oversight -- but it does a quick rough filter through things so you're not wasting time on the little stuff, allowing you more attention to find the glaring holes.