What I suggested would allow it to fix the issues. Just not write a test that was directly usable as a security exploit.
replyThis doesn't stop attackers from being able to leverage the analysis. But it does make the tool more useful for defenders than attackers. Which is the best that you can hope for from a useful tool.
It hides the issue a bit. But if you ask for atomic security fixes and then stare at the diffs you have your vulnerability. There is just a bit more friction involved in the vulnerability => exploit path, but the root cause is unfixed.
replyI think it even might be possible to route the isolated fix somewhere to automate that last step. Maybe invert the diff and pass it through automated code review for example, see the reasoning when the llm flags the change as dangerous.
> What I suggested would allow it to fix the issues. Just not write a test that was directly usable as a security exploit.
replyIt will be pretty obvious what are security issues in that case - i.e. all the code changes that don't have corresponding tests.