undefined

points

[-]

When people have gotten excited about having so many tests, I've asked them what their plan is when 500 of their 1000 test suddenly stop passing. I've always favored 50 tests I know are meaningful, where any failure is something to respond to, over 500 tests with a bunch of overlap and edge cases I'm not sure matter. But now people aren't even thinking about that, they're just letting the agents make all those decisions.

I think this approach will continue to work for some people and orgs, but others are going to hit walls where their agents can't manage the mountains anymore. It's going to be interesting to see how people and orgs respond when that happens.

by CharlieDigital6 hours ago|

parent|

[-]

    > I've asked them what their plan is when 500 of their 1000 test suddenly stop passing

That's not a terrible thing; would indicate that some shared behavior is broken and -- assuming well structured code -- hopefully easy to isolate the behavior.

Bigger problems to me are more practical: 1) large tests waste context both in and out, 2) large volume of tests waste CI cycles which are becoming the bottleneck.

by mejutoco5 hours ago|

parent|

[-]

IMO the solution to too many tests is to encode more things in the type system. If you can encode constraints like the length of an array in the type system you know it is correct if it compiles, and that is one less run-time test to run.

by CharlieDigital4 hours ago|

parent|

[-]

Agree, but this is not a pattern that is emergent from any coding agent I've used; it is only the result of explicit guidance to structure code in specific ways + relying on platform level analyzers.

But more specifically, I usually see the explosion of test code in integration tests where there is heavy setup of entity graphs.

by StilesCrisis6 hours ago|

parent|

prev|

[-]

Back up to the point where all the tests were passing, of course? Then debug.

by mejutoco5 hours ago|

parent|

[-]

or git bisect until you get the exact commit that introduced it

by crnkofe6 hours ago|

prev|

[-]

Every senior I've ever worked with at some point embraced the idea of minimalism. Produce minimal changes with a minimal surface. Worked well at startups and especially well on large spaghetti legacy projects. Seeing all of a sudden now how suddenly number of kLocs, PRs, token maxxing, number of tasks in progress is again a metric with which non-engineers brag about on engineering conferences feels like the industry has reverted to some braindead pre-infant state. I guess all mistakes need to be relearned at every hype cycle.

by CharlieDigital6 hours ago|

parent|

[-]

This doesn't solve the issue with "too many tests", but one of my teams addressed this to some extent by writing "blueprints" for test scaffolding that specifically aims to surface the scenario under test.

https://github.com/zeeq-ai/zeeq-biome

Was initially hand written as part of our codebase before AI, just extracted it as MIT, OSS since there's no secret sauce here. But really cuts down on test verbosity and makes it much, much easier to read, review, and evaluate tests.