It hides the issue a bit. But if you ask for atomic security fixes and then stare at the diffs you have your vulnerability. There is just a bit more friction involved in the vulnerability => exploit path, but the root cause is unfixed.

I think it even might be possible to route the isolated fix somewhere to automate that last step. Maybe invert the diff and pass it through automated code review for example, see the reasoning when the llm flags the change as dangerous.

> What I suggested would allow it to fix the issues. Just not write a test that was directly usable as a security exploit.

It will be pretty obvious what are security issues in that case - i.e. all the code changes that don't have corresponding tests.