undefined

points

[-]

If you're an amazing, top 1% software engineer, and you switch jobs to become a penetration tester, you're not going to be top 1% day one. There's knowledge specific to penetration testing you'll need to research and absorb and practice before you're good.

For the models, it's always day one. They don't learn from experience. Whatever they know after training is all they'll ever know. So if you don't teach them penetration testing in training, they won't know it. Maybe they could theoretically derive everything from first principles, and so could a top 1% software engineer, but that's a very slow and inefficient way to do things.

by nicce2 hours ago|

parent|

[-]

> Whatever they know after training is all they'll ever know.

Feeding enough source material as context can absolutely make them better.

by Tanjreeve59 minutes ago|

parent|

prev|

[-]

Having something dumb that just cycles through the state of the art for exploits is already absolutely incredible security tooling (if people bother to make it into a process which I'm less optimistic about)

by ByteAtATime13 hours ago|

prev|

[-]

I have essentially no knowledge in this field, but another concern may be that if the model has minimal cybersecurity knowledge, it may also implement insecure systems as a side effect.

by zamalek12 hours ago|

parent|

[-]

My initial thoughts too, it's going to write insecure software. I recall early LLMs having problems with trivial stuff like SQL injection.

Definitely going to give this one a skip.

by zozbot23417 hours ago|

prev|

[-]

> For a model it doesn’t seem that far fetched to make the leap from “I have the knowledge of what hacking is” and “I know how computer systems work” to just being able to exploit stuff anyways.

You'd be surprised. Cyber offense is a very narrow domain, especially given the extensive mitigations that are being used in modern systems. Leaving it out of post-training tasks unless you're specifically interested in it is the no-brainer approach.

by besterman2316 hours ago|

parent|

[-]

Perhaps the gap is just too wide, but to use a more simple analogy. I see this like someone knowing everything about biology, simple mechanics, metallurgy, and human history but never hearing about an axe then assuming that person would never come up with the concept, use case of, or means to create an axe.

by croniev3 hours ago|

parent|

[-]

What confuses me about this analogy is that it seems to imply this is a matter of a knowledge-base and logical inferences. But the whole reason why LLMs are so successful is that the logical approach was abandoned in favor of a statistical one (and huge amounts of data, compute, and weights). The argument is precisely, that what is not in the training does not find its way into the probability distribution. To claim that LLMs reason is a very far stretch that I don't feel comfortable to let pass.

by jamienk16 hours ago|

parent|

prev|

[-]

Will there be a book of secret knowledge that only the initiated will be allowed to see? And they will all swear an oath to never input it into an AI system?

And then we'd also do that for all kinds of potential terrorism, sex-crimes, political subversion, and evil advocacy?

It would be the "restricted section" like in the Hogwarts library?

by zozbot23416 hours ago|

parent|

[-]

Post-training is not about "secret knowledge": it's literally about training the model in a RL environment on very specific tasks (including, potentially, cyber exploitation). It's just as compute-intensive as pre-training these days, if not more (given that several checkpoints of the same pre-trained model are being released, showing sizeable gains in real-world capabilities for non-trivial productivity tasks).