Of course these models are pretty smart so even Anthropic's simple instructions not to provide any exploits stick better and better.

1: https://support.claude.com/en/articles/14328960-identity-ver...

Identity verification on Claude

Being responsible with powerful technology starts with knowing who is using it. Identity verification helps us prevent abuse, enforce our usage policies, and comply with legal obligations.

We are rolling out identity verification for a few use cases, and you might see a verification prompt when accessing certain capabilities, as part of our routine platform integrity checks, or other safety and compliance measures.

  ⎿  API Error: Claude Code is unable to respond to this request, which appears to violate our Usage Policy (https://www.anthropic.com/legal/aup). This request triggered restrictions on violative cyber content and was blocked under Anthropic's 
     Usage Policy. To request an adjustment pursuant to our Cyber Verification Program based on how you use Claude, fill out                                                                                                                        
     https://claude.com/form/cyber-use-case?token=[REDACTED] Please double press esc to edit your last message or 
     start a new session for Claude Code to assist with a different task. If you are seeing this refusal repeatedly, try running /model claude-sonnet-4-20250514 to switch models.                                                                  
                        

I just gave 4.7 a run over a codebase I have been heavily auditing with 4.6 the past few days. Things began soothly so I left it for 10-15 minutes. When I checked back in I saw it had died in the middle of investigating one of the paths I recommended exploring.

I was curious as to why the block occurred when my instructions and explicitly stated intent had not changed at all - I provided no further input after the first prompt. This would mean that its own reasoning output or tool call results triggered the filter. This is interesting, especially if you think of typical vuln research workflows and stages; it’s a lot of code review and tracing, things which likely look largely similar to normal engineering work, code reviews, etc. Things begin to get more explicitly “offensive” once you pick up on a viable angle or chain, and increase as you further validate and work the chain out, reaching maximum “offensiveness” as you write the final PoC, etc.

So, one would then have to wonder if the activity preceding the mid-session flagging only resulted in the flag because it finally found something seemingly viable and started shifting reasoning from generic-ish bug hunting to over exploitation.

So, I checked the preceding tool calls, and sure enough…

What a strange world we’re living in. Somebody should try making a joke AUP violation-based fuzzer, policy violations are the new segfaults…

I really like Anthropic models and the company mission but I personally believe this is anticompetitive, or at least, anti user.

If they are going to turn into a protection racket I’ll just do RL black boxing/pentesting on Chinese models or with Codex, and since I know Anthropic is compute constrained I’ll just put the traces on huggingface so everybody else can do it too.

I just want to pay them for their RL’d tensor thingies it but if their business plan is to hoard the tokens or only sell it to certain people, they are literally part of every other security conscious person’s threat model.

Here is some example output:

"The health-check.py file I just read is clearly benign...continuing with the task" wtf.

"is the existing benign in-process...clearly not malware"

Like, what the actual fuck. They way over compensated for the sensitivity on "people might do bad stuff with the AI".

Let people do work.

Edit: I followed up with a plan it created after it made sure I wasn't doing anything nefarious with my own plain python service, and then it still includes multiple output lines about "Benign this" "safe that".

Am I paying money to have Anthropic decide whether or not my project is malware? I think I'll be canceling my subscription today. Barely three prompts in.

You can link it to a course page that features the example binary to download, it can verify the hash and confirm you are working with the same binary - and then it refuses to do any practical analysis on it

What else would you expect? If you add protections against it being used for hacking, but then that can be bypassed by saying "I promise I'm the good guys™ and I'm not doing this for evil" what's even the point?