We will hopefully get over this wave and be able to move forward with much more secure software, but it's crazy right now.
I’m not saying that’s right, I’m saying that’s how a lot of businesses think.
Bottom line: it's a good thing. My software will be better after that. Even if it's currently not good for my ego (and my health). But I'm glad for all the tools that help me improve quality.
And sure, I can just ask LLMs to fix the bugs, which works well half of the time, and produces bad slop in the other half. This is where human expertise and judgement is still necessary to avoid letting our code bases drift into LLM spaghetti hell.
While I did fix most of the things because while they might not have been exploitable they were bugs waiting to happen, in all but one case I haven't been very impressed by any of the LARGE IMPORTANT SECURITY HOLES that were found.
But that doesn't matter for me. It shows me real bugs and less than 1% false positives. Some bugs are truly nasty and urgent, most are trivial and not exploitable at all, but they're bugs and so I'm fixing them.
Mythos isn't mythically superb for being the best LLM model, but only for not having these annoying guard rails that make all others randomly refuse to cooperate when you need them the most. Whoever invented guard rails - please stop! - they just slow the good guys down in the race.
If you don't mind answering, how come? Is it specifically that you dislike how abusable your code was, or something else?
> This has been stressing me out
> the bugs follow me in my sleep
Sounds like you need some vacation.
i would say the recent slew of vuln reports/breaches making the news rounds are doing work!
This is the silver lining i suppose!
What makes you think that AI trained on slop is suddenly going to produce quality code?
But any kind of mistake, any kind of ambiguity and you'll be in for a wild ride. It usually takes a while for the rot to set in. I've had to revert - more than once - to many commits prior to undo a bunch of work that had me completely puzzled as to what the reason that particular fork was taken to begin with. Usually I end up figuring it out but definitely not always. In some cases this kind of wild modification fed back on itself until the situation was beyond rescue. I'm a bit quicker with such reverts now than in the past, for the first year or so I would try to save what could be saved, now I just restore and try again.
But if you're not sharp when evaluating what gets done there is a good chance that you'll miss it and then it can pop up way later.
What I've noticed as correlation (but I'm not sure if that is the root cause) is that non-local effects and stack effects are particularly hard as is everything that requires a working model of the world.
edit: this comment seems to make people uncomfortable.
Harness, prompt etc etc are all peripheral IMO, though people like to think their super long prompt, custom harness and MakeNoMistakes.md skills are what contributes to their special skill with AIs, but really models are just improving.
You can just trust the AI to generate a perfect answer I suppose (and why are humans slowing it down at all if so?), but if you don't expect that yet you have to put in the work.
And on the other side, I have seen AI produce exceptionally efficient, productive and effective code - the opposite of slop - when it is treated like a junior developer who will do any idiot thing, except when it is mentored by an actually experienced, professional developer.
The source of slop in AI code is not AI, its not the training data - its in the actual workflow of the human involved in its production.
If you just turn it loose on a codebase and ship it when it passes the tests you're asking for getting burned.
LLMs finding unpatched vulnerabilities faster than they can be reasonably patched, bringing all other software development to a halt.
If you consider it to be productive work, it's a massive productivity boon happening right now. Of course, product managers might disagree...
Imagine that the whole city has to install iron doors and window grilles, because wooden doors and glass windows are not safe enough anymore.
That's a lot of work, but is it productive in the end? Probably not.
With AI it's the same. World was fine with lax security approach before. Today it is not fine. Tomorrow we might need to prove every code line before shipping, will that be more productive? Hell, no.
> That's a lot of work, but is it productive in the end? Probably not.
Your imaginary scenario defines the solution to be pointless. If that same city was subjected to weekly riots smashing windows & doors, and people regularly died, stopping "productive" work to install things that keep people alive and productive? That's productive.
https://en.wikipedia.org/wiki/Parable_of_the_broken_window
Basically, if you accept this line of reasoning, then someone going around breaking windows and someone else going around repairing them can be seen as good because it creates jobs. But that would be nonsensical and create poverty. That's why economists focus on wealth creation not job creation.
Fixing security bugs is only needed because people exploit them. It creates work but it doesn't create wealth.
Protecting yourself from bad actors also doesn't create wealth but it protects wealth you already have.
You know cybercrime and APT groups are not bored kids in their parents basement, there are groups that are running automated tooling to find vulnerable companies and if possible take all their wealth.
Just look at Binance of course, but if you are small company there are also people targeting those especially with AI assistance nowadays.
it is distinct from question whether it is productive to defend from evil people
evil people + protecting from them is obviously net negative!
but it is not changing that of evil people are present then reasonable level of protection from them is needed, and setting it up is productive
One way to think about it is that production allows you to pay for stuff. Not necessarily directly but by way of trading. It is the output you use to buy things like security.
Consider an apple farm. What does the apple farm produce to pay its bills? Apples. They could also dedicate resources to pears, &c, and that would add to their production -- to the goods they produce, the goods with which they (indirectly) purchase tractors, fence posts, &c. They could also dedicate resources to security -- more fences, &c -- and this may even be prudent. Maybe thieves are getting taller. This doesn't really add to production, though; at best, it maintains production at the status quo ante (assuming the fence is effective).
The software is better in the end but not all of it will be productive.
If that same city was subjected to weekly riots
That's productive.
The point it, this was not happening before. LLMs made cheap cyber attack possible. But it didn't make the implementation of fixes correspondingly cheap. So now everyone needs to spend extra money/effort to install grills to cover where LLMs fall short.
For postgres, this absolutely makes sense though as long as people are willing to shell out the profits from LLM use to support postgres development. That is one way this could workout for everyone..
Then they have a pikachu face when someone smashes their car window to steal something worth $50 they left on front seat because “who would do that?”.
What is even better thief will most likely pawn this for $10 but for him it is effortless $10.
Obviously this doesn't work on the internet since most hacking comes from overseas out of our control. Which makes me wonder if there is a possible future where we just block packets coming from countries that don't lock up the hackers.
Not obvious to me. I am generally a fan of Cyberspace manifesto and the free Internet, I think it's great people can reach outside their ingroup and talk to strangers and ostensible enemies.
But, that doesn't mean we should give up physical security, the guarantee that the connecting party comes from a certain physical location (state, company). I think allowing connections from anywhere by default is fundamentally flawed security architecture, for certain things.
The algorithm is: if the threat model is valid, there are vulnerabilities and you either have some software everybody knows about or a gang of hackers specifically targeting you, you should patch.
The part of the industry that profits from security theater will want you to patch no matter what, even if your business goes down in flames due to downtimes and new bugs introduced by patched components. They won't remind you to assess the threat model first.
It becomes productive only once your business stops for days because somebody brought it to a halt by breaking into the servers and deleting them.
You have a perverse view of product ivity
I don't think that's true.
In medicine, science often doesn't know how specific drug really is working. They just do a lot of tests and if these tests are positive enough, they ship it. I think it's good analogy. Biology is very complicated and it's out of our grasp to truly understand how various parts of it interact with each other on so many levels. But if something helps, it's stupid not to use it just because we don't fully understand it. Sometimes it bites. Sometimes it bites 30 years later. But we still follow that approach.
Software is similar. The complexity of the systems we build is enormous, when we're taking in account all the moving parts. I just read that someone emits radiowaves toggling memory bit and ex-filtrating information. How can you account for that? Probably you can't. But we still need software for good things.
Some people will much appreciate the holes in Postgres that are being plugged. Many more will be bored with patch releases until new features rollout.
A software affecianado appreciates patches, and might do well to teach laypeople the benefits of a good defense.
This is one of the least talked about and most useful features in any harness I would say, the review feature. I have never not found issues (either security or implementation) running it, no matter the code base or language.
Also, in the article they mention a significant number of patches for older versions as well.
It's not that the LLM code is that great, but that many overestimated the human code and the human review.
I think comparing the resources they spent at it would be a more apt comparison.
AKA if you tasked 150 mathematics departments to only focus about that problem, and stop doing things like teaching and researching anything else, you might get close to the amount of resources that these companies put into solving the problem to get a press release.
What they are really good at is writing boilerplate and reading API documentation / example code. Avoiding subtle errors is not what they're good at in my experience.
Also: most tests are positive tests. Tests rarely observe the negative space: that which is there but shouldn't be there.
looks at particledb
I wouldn't be shocked if in 1-2 years non-security vetted software released into the wild will be hacked on arrival, like Windows XP without service packs used to be.
I am not doing it anymore.
Human-written code isn’t inherently buggy, and LLM-written code isn’t inherently bug free. What is irresponsible is shipping code you don’t understand, and that will only increase with LLM-written code. A major reason why so much of the world’s software sucks is precisely that we think of it in terms of what’s “economical” and not what’s good or right. What we need is to slow the fuck down and stop treating every facet of life like a transaction. Software is filled to the brim with dude bros who “move fast and break things” and actively exploit people as their business model and somehow are called “the brightest minds of our generation”. Our priorities aren’t right.
And even if the impossible occurs and LLMs hack everything that's connected to the Internet, so what? That would be inconvenient but it wouldn't cause doom. People would muddle through and eventually fix the critical problems, just like they always do.
You would think so but medicine has faced this paradox since the development of modern testing technology.
You would naively think more testing is always better. More sophisticated practitioners then talk about Type I vs Type II errors and the problem of false positives.
But even if you were able to come up with tests that had 100% accuracy, there are times when you don't want to run that test on someone. The key distinction is that people only ever die of one thing and having a tumor that doesn't kill them is often better than risking the surgery to remove the tumor.
Similarly, there are some bugs you don't want to find since the harm from the people it affects is smaller than the opportunity cost of fixing it. Ignorance of the bug is better than knowing about it and having to make an affirmative decision not to fix it.
But what applies to humans doesn't apply to software. First, to pursue the metaphor, codebase can definitely die/be compromised/crash because of multiple reasons at the same time; second, there is virtually no risk, out of man-hours, in fixing a bug.
Logically, a bug that prevents someone from doing something and a missing feature that prevents someone from doing something are equivalent, they shouldn't be thought of as two separate activities but should be evaluated via the same framework. The list of all defects in a piece of software is infinite so it becomes impossible to erase all defects. It's about prioritizing with finite resources.
> You can't predict the harm from a small bug when AI can chain complex exploits and when your software is used in mission critical applications as a data store.
Yes, any bug can theoretically be part of an exploit chain, just like any tumor detected can theoretically be the tumor that kills you. This is why the detection paradox exists. Once you become aware of the tumor, you face a duty to try and mitigate it. But we know from autopsies that people die with all sorts of tumors that didn't kill them.
Similarly, if we bothered to do autopsies of software products after sunsetting to discover a comprehensive list of CVEs, we'd discover a ton of bugs we were better off not knowing about because they ended up not making an impact but we would have diverted resources if we knew about them.
This is not an argument that we should remain totally ignorant of all bugs, but a refutation of the "common sense" belief that more knowledge is always better.
I guess it really depends on the severity of the defect. Would it allow some really significant exploit? -> Important to fix.
Does the defect only prohibit one very esoteric use-case which basically never comes up (and was thus never reported by a human user and only now that we have an infinitely patient machine checking every completely unlikely code path is it detected) -> more productive use of your time to develop a new feature which would be used by more people.
Firstly, the cost of fixing a bug once you know it exists is usually minimal compared to the cost of discovering that bug. This is the inverse of medicine, where the cost of testing (at least in terms of health outcomes for the patient) is typically minimal compared to the cost of treatment.
Secondly, if you have an asymptomatic illness, this does not typically make you more vulnerable (hence why it's asymptomatic). Like you say, you typically only die of one thing, so if you know the patient won't die of a particular diagnosis because something else is more likely to kill them, then it might not be worth making that diagnosis. But for software, you can be simultaneously vulnerable to lots of things at once, and any of these vulnerabilities can be the one to "kill" you. Some vulnerabilities may be easier to find and exploit (or do more damage) but in principle any active vulnerability can be exploited.
Thirdly, having multiple vulnerabilities makes software more vulnerable overall, because those vulnerabilities can potentially be chained together to create more serious exploits than each vulnerability would have represented on its own. To a certain extent this can be true in medicine as well, but generally it isn't.
Fourthly, we can't cure death (yet?) but we can cure all the vulnerabilities that we've found so far. There is no known upper bound on the lifespan of a piece of software, and we know that older code tends to have fewer bugs in that newer code, presumably because those bugs have been fixed over time. This is different from health issues, where a given condition may be chronic or terminal and therefore at a certain point there's nothing you can do to fix it.
Finally, ultimately in healthcare it's your body, and your decision was to what you do with it (assuming you can find a doctor willing to do what you want). If you have a condition and you want it cured, even if it is completely benign and will have no impact on your life whatsoever, then you can do that. (Again, assuming you have the resources and the condition is curable). But a software vulnerability does not affect you as the developer of the software, it instead affects your users/clients/customers. Therefore there are different dynamics in play. If I sell you a tool and I know it could be vulnerable, but I don't try and find out what those vulnerabilities are, then you're ultimately the one on the hook when all your data gets deleted or leaked.
Boy, I wish!
> Like you say, you typically only die of one thing, so if you know the patient won't die of a particular diagnosis because something else is more likely to kill them, then it might not be worth making that diagnosis.
You can't know someone didn't die of X until after they died, that's the entire point of this dilemma.
The key thing you forgot, which makes this approach non-sensible, is the omniscience, without which you have no clue whether your posture towards a new bug-finding tool is good or not, since you don't know whether it will mostly find those bugs that are not worth fixing and the time wasted on fixing them is detrimental.
Similarly, how is ignorance better than informed decision when ignorance also includes missing critical security bugs?
This isn't an argument against all automated bug detection, just that it needs to exceed a threshold beyond just "It has found at least one real bug not found via other means" which is the seductive threshold people want it to be.
That very much depends on got you define “bug” and how you specify your product requirements. Also, you have to assume the specs are correct.
We want software that remains robust in the long term, so that even when people are able to inspect the source code, it's still solid, rather than hiding the source code and hoping people don't find holes everywhere.
"We found a bunch of ways to make our product safer and better. We need more time to ship a higher quality product."
PICKET SIGNS
No, they were not. The problems they faced were absolutely real, but the solutions they chose to attempt were stupid and counterproductive. Which is why they lost so hard, and it was not even a contest.
Socialism movement that refused to use machines and factories would inevitably lose to capitalists that embraced them.
Socialism movement that refuses to use AI will inevitably lose to capitalists that embrace it.
> In the end, I don’t think that the delay in shipping PostgreSQL 19 is particularly concerning. It is not a long delay in the grand scheme of things, and it emerges directly from the project’s concern for quality.
So unless it gets outlawed, I don't see people opting out of it
The article also concludes that overall, yes, finding more bugs is a good thing. Basically saying it’s the LLMs “fault” that the quality is improving, alongside a slight delay to the release.