These statistics and safeguards are pretty much meaningless, because it's completely trivial to get around them. You can prefill and edit the model's responses/thinking and force it to agree to virtually anything even with unmodified weights, and if you don't want to bother with jailbreaking and can host it yourself it's essentially a one-man weekend project to uncensor the model and have it agree with anything you want by default.
replyGuardrails on open-weight models is security theater purely to shield whoever released them from a potential PR nightmare.
I was trying this with Gemma 4 before and it wasn’t as easy as I expected.It would refuse even after long prefills.
replyIt's unfortunate, because the security theater performed here detracts from the model's overall cybersecurity performance. And uncensoring has another small hit to performance.
replyYea, the problem with maliciousness is intent.
replyMuch like having a hacking tool doesn't mean you're doing anything illegal, attacking a machine doesn't necessarily mean you're committing a crime. I can hack the living shit out of my local network if I want to.
deleted
replySomehow that statement just screams that our model is not useful for security work
replyNobody cares about that number, because refusals don't lead to company-ending news headlines.
replyIt is kind of comical how things that previously would have been obviously true are now only statistically true
reply"Our product won't hack you 96.6% of the time"
"The model failed to refuse a cyber request when it should have" != "the agent hacked someone". For example, refusals also apply to just asking the model how to do something, without extending it any agency.
reply