That said, if we were a closed source project, I don't know that our own tooling would have discovered all the issues using LLMs as we are seeing more and more creative ways of people exposing exploits that I never would have thought to put into the LLM prompts.
If all you had was an API endpoint, you wouldn’t realistically have any way of figuring out how the internals work and finding obscure exploits.
A 1000 of us regular devs scanning a piece of code is not as effective as a single review by, say, Tavis Ormandy. So "enough eyeballs" is only "enough" when one of those set happens to belong to a Tavis O. or the likes.
Not to mention, this assumes a 1000 of us even look at the code, which is statistically never true. Approximately 100% of us just install the software without looking at a single line of code. Hence the code auditability aspect of open-source software never really applied. And hence, "supply chain attacks."
Unfortunately, this misguided "law" was taken as gospel by many, and as you found out, still is.
Instead they court martial anyone who talks about the designs. Because obviously, telling everyone every aspect of your security isn’t actually more secure. How would you know how to do a 24-step process to exploit something if you have no idea what code is running on their server?
Imagine trying to build a cheat for CS:GO if you had no access to the game and could only play it via a remote connection. Good luck.
It's true but the problem is assuming, just because something is open source, it has received sufficient eyeballs. AI now gives us near unlimited eyeballs so, if we survive the bumpy transition, open source should indeed become very secure compared to closed source.
Except we won't get the expected benefit, because all those eyeballs are attached to a very small number of different "brains" (LLM models.)
The original saying depends on an implicit assumption that eyeballs are proportional to minds, and that among those varied independent scans, different people will catch what others miss.
In other words, we shouldn't assume 5.5x review thoroughness when we hire 11 QA eyeballs but they all belong to Bobby the Beholder.
My point was more that very few brains were looking deeply at the long tail of open source code/projects (other than the maintainers of the code themselves); and for the marquee projects that did get proportionally more varied eyeballs (e.g. postgres), it seems even that level of attention was insufficient for code deep inside the project's core.
LLMs change this. They make it much easier for maintainers and motivated users to amplify the power of their eyeballs.
I don't understand how you get to that conclusion.
Do you think that an open source project will receive more AI eyeballs than a closed source project?
Net net, yes.
Linux kernel will get an insane amount of tokens burned on analyzing it. Most other open source, not so much.
Assuming offensive and defensive capabilities are similar, and incentives scale roughly with project size but start from a non-zero value for attackers, I'd expect larger open projects to be well defended by plenty of motivated people, and smaller ones less so (depending on maintainers). Closed source will mostly just depend on maintainers.
Also assuming closed source is less exposed to AI auditing overall, I'd guess large open projects end up more secure than large closed projects, but small open projects possibly less secure than small closed projects, since attackers might have more incentive to target them than defenders do to audit them.
I'd lean more towards systems that only install diamond-hardened software at all - once you have one project in a class of software that has been analyzed to death, that's the one everybody is going to use.
The vast majority of the development of all projects is going to be (already is) LLM-assisted, so they're going to get analyzed every time they're touched. The odds that huge bugs are going to be lying everywhere to be exploited are bad imo.
I'm starting to rewrite an open source project just because I don't want it to be buggy anymore. I barely want to change anything about it (especially not the licensing.) All I'm doing is porting it to rust from c++ and refactoring it like crazy. I just want to make it easy to audit and extend, and hard to crash, that's it. Once I'm done, I'm already have another (buggy) project in mind to do the exact same thing to. Great UIs and usability, just thwarted by the time that the developers had to invest in them. You could think of them as almost purely security-oriented rewrites.
Using open source source software lets you share responsibility and oversight with a large number of people. But it brings with it a much larger attack surface than you'd have if you only implemented the part of that software that you need, and a much more complicated threat model for supply chain risk.
If a large number of people collaborate on open source software for something, especially if it's security or infrastructure related, that makes the project itself a much higher value target than any individual company's home grown mail server or http proxy client would be. Participation makes your vulnerabilities more public, since interacting with the public open source project, making blog posts or hiring ads mentioning it, hosting it (and thus serving traffic in a way that signals or directly tells clients what implementation it's using), bundling it with your site or binaries, etc. tells prospective hackers 1. how to find potential exploit targets for a general open source vulnerability 2. where to look for or introduce vulnerabilities to pwn a particular target.
AI auditing isn't really a silver bullet because it costs a non-trivial amount of money for each commit or PR, and scales with the number of contributors to a project and the project's size. Normally you want more of those with a software project, but in this case it makes AI auditing more expensive unless it's shared across the project and basically part of its CICD (and I don't think that's really been fully adopted yet). But then that introduces other risks like adversarial payloads where the attacker verifies that the AI security checks in place for a project don't trip when they actually send the malicious code for review, because they can run the same checks offline and tweak the payload until it passes.
Personally I think we're headed toward an outcome where only very challenging, well-resourced, and critical open source software is worth using for developers. For smaller things it makes more sense to just use AI to implement your own version of it, or fork/take inspiration from a project and mold it to fit your needs.
I agree on this part, but what's your point here? Are you suggesting closed source is better?
Also, closed source software can similarly be analysed with AI, but there's no guarantee that the company involved will deal responsibly with disclosures.
Do you have examples of people saying this? I don't think I've come across idea being espoused...