upvote
It's possible that AI is going to be better at finding and exploiting systems than at securing them. It's not really an even playing field. On the one hand you have to somehow ensure that you've patched everything when there's no definite list of everything, the other side just needs to find one overlooked way in.

At the rate of development, also, will AI developed in 6 months from now be able to break a system that was secured by todays AI?

reply
It's not possible. It generally requires more work to find an exploit than to patch the (known) exploit.

Now, you could argue that writing secure code ex-ante is harder than discovering an exploit. But you can hypothetically run the same "exploit finding" process on a first pass of code before it ships (e.g on the PR) given low enough token costs, which is where we're headed.

reply
> It generally requires more work to find an exploit than to patch the (known) exploit.

Even if that's true, one should not forget that:

When the hackers have breached something, awaits a reward.

E.g. a Bitcoin wallet, or learning something about a political competitor and getting rid of them.

(So the hackers can spend lots of ...)

reply
How do you know when the "exploit finding" pass is done? A hacker group would happily burn a month or more of tokens trying to break a critical piece of software.
reply
But are LLM patches exhaustive? Cant a new model find even more obscure bugs, including bugs introduced by the old LLM patch? Is it really realistic that we such complex projects as some of these critical dependencies are can be truly 100% bug free?
reply
If the software is too complex to understand, maybe it should be made simpler. Nginx does a million things currently but what most people need is something that parses http1 strings and sends back a file or calls a second program based on the path. Every component can be thought of in this way, with the most complexity perhaps in being sure the interfaces don't do something unexpected
reply
We dont need to be bug free, so much as the juice needs to be not worth squeeze. I think i am also making an assumption that we "level" in terms of frontier model progress at some point, and we have months or years between upgrades.
reply
>so much as the juice needs to be not worth squeeze

I wonder about that though: if the price of the squeeze collapses to 0 because of this (think of what models and prices might be like a year from now), I think I know where all those out of work mathematicians can find gigs. We are going to need to invent a layer on top of, an evolution of or some new concept a level above "computer programming" that deals with meta-organization of code.

reply
but at least sysadmins/ops still have work to do
reply