upvote

points

by themafia3 hours ago |
comments
upvoteby tptacek2 hours ago|
next
[-]
Sorry, can you clarify what you're saying here? What didn't work that well?
reply
upvoteby themafia2 hours ago|
parent|
[-]
Letting Claude get at the source code to try to find CVEs. I found it particularly entertaining that after finding none it just devolved to a grep for "strcat."
reply
upvoteby tptacek1 hours ago|
parent|
[-]
Oh, I see. No, you're wrong. That's absolutely not what it did and not at all an accurate way to sum up what it found.

This isn't a complete rebuttal to your argument but I'll note with irony that we're commenting on a thread about a FreeBSD kernel remote that Claude both found and wrote a reliable exploit for (though people will come out of the woodwork to say that reliable exploitation of FreeBSD kernel remotes isn't much of a flex).

Here, from the exact tranche of vulnerabilities you're saying was just a "grep for strcat", are the Firefox findings:

https://www.mozilla.org/en-US/security/advisories/mfsa2026-1...

We're getting to a point, like we did with coding agents last year, where you can just say "I believe my lying eyes". Check out a repository and do Carlini's "foreach FILE in $(sourcefiles); <run claude -p and just ask for zero days starting from that file>". I did last night, and my current dilemma is how obligated I am to report findings.

reply
upvoteby themafia1 hours ago|
parent|
[-]
It's from the link I posted. Claude's own team in January trying to do exactly what you suggested and ending with results that are less than promising. It's their blog. I assumed it represented the pinnacle of their research.

We're getting a point where anecdotes are being used in place of reason. I'd think you want to ask "how many bug bounties are earned by humans vs AI assistants?" If there's money to be made in finding 0-days then shouldn't there be ample evidence of this?

reply
upvoteby tptacek1 hours ago|
parent|
[-]
You can see now that you assumed wrong.
reply
upvoteby joe_mamba2 hours ago|
prev|
[-]
How did you managed to get it to do that? When I gave it instructions to use Ghidra MCP to look for vulnerabilities in a windows driver on my local machine it refused saying it's not allowed to do pentest activities even if sandboxed to your own device.
reply
upvoteby tonyarkles2 hours ago|
parent|
[-]
Not who you were asking and not explicitly looking for vulnerabilities... I have gotten a ton of mileage from getting Claude to reverse engineer both firmware and applications with Ghidra and radare2. My usual prompt starts with "Here's the problem I'm having [insert problem]. @foo.exe is the _installer for the latest firmware update_. Extract the firmware and determine if there's a plausible path that could cause this problem. Document how you've extracted the firmware, how you've done the analysis, and the ultimate conclusions in @this-markdown-file.md"

I have gotten absolutely incredible results out of it. I have had a few hallucinations/incorrect analyses (hard to tell which it was), but in general the results have been fantastic.

The closest I've come to security vulnerabilities was a Bluetooth OBD-II reader. I gave Claude the APK and asked it to reverse engineer the BLE protocol so that I could use the device from Python. There was apparently a ton of obfuscation in the APK, with the actual BLE logic buried inside an obfuscated native code library instead of Java code. Claude eventually asked me to install the Android emulator so that it could use https://frida.re to do dynamic instrumentation instead of relying entirely on static analysis. The output was impressive.

reply