The key seems to be extensive integration/end-to-end tests with gold standard assertion data. Heck, even just saving off the json from API endpoints and using that as a reference to compare after changes works pretty well. Spin up a database backup with that static starting point, run actions, compare state afterwards.

These types of high-level tests are frustrating beyond belief to humans due to their lack of specificity, but with the agents, they don't get annoyed investigating possible regressions from non-specific signals.

They also aren't as painful to maintain as one would think, because a regression flagging test can be traced by the agent and represented as the business rule that was violated. I've found recent models to be really excellent at discerning a true regression from an outdated test assertion, especially if they are able to trace the failing test back to the PR and work ticket that built it.

Management still pushes for more ai and will rather hire more heads to "handle" issues.