upvote
LLMs and particularly Claude are very capable security engineers. My startup builds offensive pentesting agents (so more like red teaming), and if you give it a few hours to churn on an endpoint it will find all sorts of wacky things a human won't bother to check.
reply
as a pentester at a Fortune 500: I think you're on the mark with this assessment. Most of our findings (internally) are "best practices"-tier stuff (make sure to use TLS 1.2, cloud config findings from Wiz, occasionally the odd IDOR vuln in an API set, etc.) -- in a purely timeboxed scenario, I'd feel much more confident in an agent's ability to look at a complex system and identify all the 'best practices' kind of stuff vs a human being.

Security teams are expensive and deal with huge streams of data and events on the blue side: seems like human-in-the-loop AI systems are going to be much more effective, especially with the reasoning advances we've seen over the past year or so.

reply
Every conversation I've been a party to has been premised on humans in the loop; I think fully-automated luxury space vulnerability research is something that only exists in message board imaginations.
reply
I am seeing something closer to the opposite of skepticism among vulnerability researchers. It's not my place to name names, but for every Halvar Flake talking publicly about this stuff, there are 4 more people of similar stature talking privately about it.
reply
People use whatever tools are the most effective and they have plenty of incentive not to talk publicly about them. I think the era of openness has passed us by. But why does stature matter anyway? If I look at chromium or MSRC bug reports, scarcely any of the submitters are from Europe/US and certainly don't have anything resembling stature. That guy hasn't done anything of note in the field in a long time from what I know, he's kind of boomer (you too, no disrespect).
reply
Vulnerability research is exciting and profitable, but it has three problems. First, it's mentally exhausting. Second, the income it generates is very unpredictable. Third, it's sort of... futile. You can find a 1,000 vulnerabilities and nothing changes.

So yeah, it's the domain of young folks, often from countries where $10k or $100k goes much farther than in the US. But what happens to vulnerability researchers once they turn 35? They often end up building product security programs or products to move the needle, often out of the limelight because they no longer have anything to prove. And they're the ones who write checks to the young uns to test these defenses and find more bugs.

The fact that the NSA or the SVR now need to pay millions for a good weaponized zero day is a testament to this "boomer" work being quite meaningful.

reply
Claude Opus 4.6 has been amazing at identifying security vulnerabilities for us. Less than 50% falae positives.
reply