upvote
AI is securing the internet. This is great. You should assume professional hacking gangs and state actors discovered (at least some of) these vulnerabilities months and years ago.
reply
Both are true. The problem is security is a much harder job than attacking.

Worse patching adds instability into a system, especially when you don't have much time to test.

Also in the past attackers were willing to sit on exploits longer and not do things with them, now they have to act fast before someone else finds it, leading to more events faster wearing your security and patching teams out, and making users life suck.

Things are going to get much worse before they get better.

reply
Yes, things will get worse before they get better. Attackers have the advantage initially and get to pick a lot of low-hanging fruit, but the advantage will shift to defenders as code becomes more heavily tested and verified.
reply
One hopes, but the models are also getting smarter and deeper too. I don't know if anyone really has a clear vision of when the future plateau state is where equilibrium is achieved on this.
reply
I find this kind unreasonable: software has finite bugs, no matter how smart your attackers are. Current CVEs are pretty subtle, and I see no reason to think that they're actually low-hanging fruit.

I guess you could make an argument based on some estimate of how fast exploitable bugs are newly introduced. That also seems dubious.

reply
This is the correct view. Look at blockchains.
reply
Question is how big is that finite number? One a month for the next 40 years helps no one.
reply
>Worse patching adds instability into a system, especially when you don't have much time to test.

Survival of the fittest I'm afraid will build security here. The teams that barely ever patch or don't have "time" for maintenance are going to get burned. The "we haven't updated in 7 years so things are stable" teams can't survive.

reply
I once did work for a company that hadn't updated or patched anything in over 4 years. Systems with 1500+ day uptimes. Totally insane.
reply
deleted
reply
Attack the attackers.
reply
Except, human in the loop in control - blue team is not defending red team fast enough that most the CVSS v4.0 Critical CVEs aren't easily patched in coordinated fashion. Many times, almost impractical or impossible to patch at all, with decades of intertwined dependency tree and their ecosystem.

Collectively, we owed about 40+ years of tech debt on non-secure code (e.g. automotive industry, industrial environments) that will be torched in the next few years. Luckily, AI will generate code faster enough to make that plausible.

reply
AI provides the capabilities to secure the internet, maybe, but the internet is secured when site owners choose to secure it.

That's not happening at nearly the same pace, both for assymetry between what it takes to break something and what it takes to fix it, but also because that fix must navigate bureaucracies. Also, the leaders need to actually care, both of which are hurdles. Bureaucratic navigation and consensus finding has not sped up at all with AI.

Compare that to independent script kiddies with $600/mo Claude subscriptions able to find 0-days in an hour and write a worm to exploit it in four.

reply
Yes but the barrier is infinitely lower. If you were used to running servers just for fun, that kind of threat was probably not too much of a concern. Now it very much is because it can be exploited at a much larger scale.
reply
Yep. In recent discussion this came up, and my analogy was that LLM will cause a similar effect to widesread proliferation of TLS. It was easy to snitch on people before TLS had been deployed everywhere. Not so much after that.
reply
> > AI is burning down the internet.

> AI is securing the internet.

No doubt there are points of view where either one, or the other statement, or both, or neither, are true. You can really go full Schrodinger and put a probability distribution on everything.

But in this day and age, whatever is happening, the better question is:

Is it good for me and, beyond me, for the world in general?

reply
AI is also making it much easier to attack, likely more than it's helping secure things
reply
it's a timing thing - eventually we'll reach an equilibirium

it was previously expensive to attack, so people spent less money on defense.

It's cheaper to attack, and so people will increase their budget on defense. But it's a lagging cause & effect situation

reply
You're a nanometer away from understanding why labs have powerhouse internal models that they only allow partners to use.

Even Kimi K3 was only shipped as a lobotomized version of it.

reply
the idea that internal models can protect the Internet like this is so dumb IMHO.

A huge fraction of devices and servers are on old versions, not actively updated or managed in any way.

reply
Then the best option for everyone's security is to hack into them and kill them.

If you're not going to take care of it you've created a public nuisance.

reply
This internal models stuff remains marketing BS afa I am concerned. Smart enough to be a junior engineer and steered toward the right tools and behaviors within a context is more than enough. Places with "super models" do nothing to demonstrate they wouldn't have done better to use the resources on relative brute force within the security domain. They have no sales pitch for good results that just needed Legos and cash.
reply
Man, is it aggravating to read fantasies like this, as someone who's trying to juggle a full-time job and hosted public services for other people at the same time.

There are two internets. One of them is open, less-commercial, and largely built and maintained by people whose first job is not the work they do to maintain this internet. That group includes a ton of open source software, millions of small servers scattered around the world (virtually private or otherwise), and is generally best-effort.

The other internet is fully commercialized, often ad-driven, fully integral to the data broker economy, and centralized behind a few companies that pay as few people as possible to satisfy the occasional PR statement that, "we take security seriously and...".

The next stable equilibrium here is not "we get a secure and open internet". The next stable equilibrium here is, "we have no open internet, and the internet is now managed by just a few companies."

We're already halfway there. Google and Microsoft combined already account for somewhere around 30% to 40% of email traffic. 30% of all websites are now behind Cloudflare. A few companies dominate social media traffic. Amazon and Shopify combined are half of US e-commerce.

Every single short-response-window exploit like this just pushes a few more percentage points of traffic into a handful of companies. Cheer for that, I guess.

When I have time to daydream, it's increasingly about building a new, underground internet, and how best to disadvantage bots and BigCo conglomeration from the start.

reply
I wonder whether part of the solution is not using the same software as big tech. For them rapid development is worth extra maintenance. Things like the efficiencies of http2 over 1.1 actually matter. Maybe the rest of us should be using BSDs or different Linux distros (Alpine? Void? Slackware?), different web servers, turn off features we do not need (e.g. not disable http2, or at least not allow both 1.1 and 2 as that is clearly extra attack surface).
reply
My sites are now sat behind my own custom reverse proxy. It's missing tons of features, which I guess I can now call a security benefit, and it pretends to be nginx just because I thought that would be quite funny. I've debated open-sourcing it but I a) don't want to maintain a project right now, and b) security-by-obscurity is back in action.
reply
> When I have time to daydream, it's increasingly about building a new, underground internet, and how best to disadvantage bots and BigCo conglomeration from the start.

https://reticulum.network

reply
Appreciate your take. Question in my mind is if the less-commercial internet will have the resources to defend against attacks made easier and easier by AI.
reply
Usenet.
reply
It's brilliant isn't it?

The real problems here are that we did a crappy job in the first place and left a huge surface area of risk with the blind assumption that we didn't do a crappy job.

I am hoping to see a change in attitude to risk and quality in the industry but I get the feeling that's asking for too much.

reply
Well I didn't get into any trouble with the professional hacker gangs. The script kiddies on the other hand...
reply
There's no honor among the thieves if everyone can be a thief.
reply
One would think that the NGINX team would be banging on their code with AI agents trying to attack it for preemptive security.
reply
"Hacker gangs"
reply
What term do you prefer?

Organized Cybercrime?

Digimafia?

e2p3yHizAungrCeKum7ifrssu32lnzEnyRoGdcAo23m2dNapvoGmymid.onion?

reply
Kim Jong Un's megayacht fund
reply
Just like in the movie "Hackers".
reply
Hey, that's a piece of art!
reply
Poor security practices are burning down the internet not AI. Many people warned and got ignored, by developing a memory safe language, introduce large scale fuzzing, utilize verification tools and other things.
reply
It's like blaming people for having wooden doors when robots with chainsaws start appearing in the street.
reply
There's definitely an issue of scale, but a wooden door at least lives up to its own "contract". There's no bugs in a wooden door, everyone can see plainly it doesn't withstand a chainsaw.

A more apt analogy would be that pretty much all consumer-grade locks are very easy to pick, despite being marketed as secure. I think in this case the lock makers should be blamed, not the lock pickers.

reply
It's been a culture decades in the making.

The lock makers are us - the IT field at large. As individuals we might not be to blame, but we are still enthusiastic parts of the greater movement that's issued vast digital complexity to the world without truly understanding it.

Now we are starting to understand it more.

reply
When 85 percent of doors were installed before metal doors were available.
reply
> robots with chainsaws start appearing in the street

Please stop willing things into existence.

reply
What about ASLR and DEP? Guess they are useless?
reply
For eight years, MongoDB shipped a default configuration that left it opened to the internet with no password.

Obviously, AI makes these things easier to find and exploit, but this isn't even close to the first time that 'thousands' of hobbyist (or small business or just not very competent) server admins are going to get rootkitted en masse. I worked at medium-sized infra-as-a-service company when the above was discovered and there were definitely thousands of compromised machines just there.

reply
web scale vulnerabilities
reply
It’s for the best, IMO. We must now design every system on the assumption that every vulnerability will eventually be exploited and every bug will be found. Relying on security by obscurity or following poor security practices will be costly! In the long run, I believe this will improve software quality.
reply
AI is -> Forcing you to PAY UP your Tech Debt.

This is the sys/devops time to shine. Everyone needs to have an upgrade and an emergency plan, and everyone who was previously only doing conventional codemonkey needs to start learning more of the stack to join in on that.

reply
This is some insane framing. This vulnerability exists already. Even if there were no LLM, this would still exist and might even be actively exploited right now. We just wouldn't have a chance to know it.

I do think the world is going a little insane tho. People suddenly believe in security through obscurity because, uh, AI bad?

reply
If a vulnerability is practically impossible for a human, but possible for AI, then it's not security through obscurity. Technology just advanced past a line where new things can be exploited.

There might also be a day in the future where our current encryption methods are broken - it doesn't mean they're insecure and vulnerable today.

reply
None of these vulns was impossible for a human. There just wasn't that many humans clever enough to find them, and those that did we're busy not hacking random things. There isn't a single type of new exploit that AI has found that I know about.

We have a billon hacking experts now is the big change.

reply
This comment is very dramatic and very hype-y. I don't think this is an appropriate comment.
reply
"Cattle, not pets" has been industry best practice for a long time now. Those that don't follow it will eventually pay the price for one reason or another.
reply
This is why I worry about upgrading in place between major Fedora versions. I think every six months you really need start from a fresh image.
reply
I am having a hard time understanding this position. AI is uncovering flaws that "state-level actors" are exploiting since decades, for some of the recently found issue.

Care to elaborate why you think that sufficiently paid humans would not be able to find the flaws AI now finds, for cheap?

reply
Agents are simply faster.

Harnesses allow LLMs to build & process large stores of data to reuse later. Imagine an agent that spends 3 days breaking into company A, records what it learned during that process, letting it break into company B in just 3 hours.

Defenders win by having the best security 24/7 and must trust that other services they depend on are doing the same.

An attacker wins when they can breach any defender's security.

reply
Not AI but poorly designed programming languages ignoring integer overflows.
reply
I don’t know any systems programming language that doesn’t ignore integer overflow/underflow in release builds. IMHO this is a CPU design flaw - integer operations should carry a flag indicating if the operation is allowed to wrap or if it should generate a trap. Then there wouldn’t be a performance penalty for this and languages like Rust could have it be trapped by default.
reply
No. Memory safe languages still have integer handling issues, but the consequences are different. In a memory-unsafe language, integer mishandling often nets out to bad copies and memory corruption.
reply
I wouldn’t be sure that a Rust equivalent would be immune because it’s not hard to imagine an underflow/overflow from reaching an unsafe block that’s used for performance (the implicated TLS path would be a hotspot worthy of such a choice).
reply
This is the thing about `unsafe` blocks: they're not memory safe. It's right there in the name.

I'm not making a Rust advocacy argument. I'm just saying: it's true that integer handling issues are endemic to mainstream programming languages, but the integer handling itself is almost never the actual sharp end of the vulnerability.

reply
Isn't "memory corruption" typically the fast path to RCE? The attacker gets to overwrite a function pointer, or a regular pointer that then lets them overwrite something else, or the length field that will then cause a buffer overrun even in a "safe" language because the run-time length is now different than the one in the compile-time check, etc.
reply
Memory corruption -> RCE.

Integer handling mistake -> memory corruption -> RCE.

That's the pattern. In a memory-safe language, the integer handling error has no link to memory corruption (though it will to other logic bugs!).

reply
Zig traps arithmetic overflow/underflow when compiling in ReleaseSafe.
reply
X86 had the INTO instruction to trigger a fault if the Overflow flag is set, so not quite a per-instruction bit but more of a suffix. JO is also fairly straightforward for compilers to add on important arithmetic operations. We can probably blame C entirely for this; only in C23 are there intrinsics for overflow detection on arithmetic.
reply
Lisp or Python have a proper numeric tower and don't default to machine types. Keyword being default, fast is still possible in CL.
reply
"I don't know any decent language so let's continue using poorly designed languages". Please look at Swift, Javascript, Python.

Why would CPU designers make a trap if languages do not require it?

reply
JavaScript and Python aren’t systems languages. Swift often pays the performance penalty for trapping everywhere and hasn’t seen adoption as a systems language for that reason.
reply
How does that saying go?

"Go fast and get your things broken into"

reply
We're after a way to go fast without having your things broken into. "Language allows you to specify whether you want wrap around or an exception and hardware makes implementing that efficient" is better than both inefficient runtime checks and silent data corruption.
reply
i still remember win xp pre service pack 1 days where you could get p0wned by a worm just by leaving machine on the internet. this is nothing compared to that.
reply
Think of it as the opposite of security through obscurity: at some point the hackers won't be able to keep track of it all. They'll start paying for Jira accounts and need to hire PMs. They'll be patching servers to drive away the competition and will need experienced IT people. Someone is going to have to build apps to keep track of it all.

The circle of life.

reply
Not quite
reply
There is going to be hardships before this gets better. But it is hardship we must endure
reply
and C is the tinderbox
reply
The billions of lines of C and C++ code is like a square lightyear of agar for LLMs to find bugs in.
reply
I’m generally not in favor of government meddling with private property, but I feel like I could be on board with the NSA or similar covertly patching vulnerable machines.
reply
And your first thought was the NSA? Patching and adding their own rootkit eh?
reply
Don't you dare patching my honeypots!
reply
I'm pretty solidly in the AI skeptic camp. But adversarial use of AI to improve software security is a good thing in my mind.

I'd be more worried if the response isn't more secure development practices in better languages, but instead use of AI as a defense. Then we're truly talking about burning down the internet.

reply
At the end of the day it will be both. In fact I wouldn't be surprised if we started seeing offensive defense. That is already compromised boxes counter hacked and erased or otherwise made unbootable.

A dead box isn't part of any bot army.

reply
Tech moves too fast and if we move too fast without doing review like 5-10 times, this will keep happening.
reply
>> Tech moves too fast and if we move too fast without doing review like 5-10 times, this will keep happening.

Tech that becomes infrastructure like web servers doesn't move so fast and it shouldn't. Most people and companies should be relying on relatively tried and true software, so this phase of AI finding vulnerabilities in it should pass and things could get better.

reply
> tried and true software

Like Java. Surely anyone can't mess up simple things like base64 decoding in it...

reply
I would hope you know that no matter how many reviews, humans are fallible and miss things.
reply
AI agents are also fallible? So, what do you think we should do?
reply
Yes, AIs are fallible. Yes, humans are fallible. Nobody said differently.

That's orthogonal to the point I'm making which is that no amount of reviews is going to make every single bug go away. Because, once again, humans and AI are fallible.

reply
The difference is that the AI doesn't get tired so you can review everything.
reply
Except that it's very far from guaranteed to find everything, and especially everything that humans would find easily.
reply
Really? AI is catching all the easy bugs I’d catch, and way more subtle ones as well. Maybe it was the case they’d miss obvious things often a year ago, but that’s not today’s frontier models.
reply
Oh, if it's catching all the "bugs you'd catch" that's sure proof it will always catch all the bugs that people would catch....
reply
Yes, we both posted anecdotes, very astute.
reply
Well, I only said that it's far from guaranteed to find everything, while you seemed to argue that your anecdote guarantees it
reply
Every now and then, I think about the old demonization against encryption… and how it completely went away.

Sure, there are niche things. You can’t use US ciphers in China and vice versa.

But overall, one day, the intelligence communities world wide just decided to not raise a fuss about everyone having solid cryptography.

It makes me think if we did find out that every server out there had a route kit… it would make that part make sense.

reply
It's going to be rough sledding for a while, but, so long as you're on FOSS, at some point, the bugs and vulns should run out, and best practices trained into AI should hopefully reduce new bugs being introduced.

All bugs are shallow given enough eyes, and the marginal cost of eyes is now ridiculously low.

reply
AI is just showing how foolish using such a unsafe language like C to write code that connects to every other computer on the internet was.

Imagine a alternate reality where Ada replaced C by 1995. We would be in a MUCH better place.

reply
Unlikely. An Ada compiler would be an expensive additional purchase, a C compiler usually came as part of the installed copy of UNIX.
reply
Also in the few years prior to 1995 most computers that could run a C compiler weren't big enough to run an Ada compiler. Heck most computers today can't run the rust toolchain.
reply
The bigger problem is lack of liability: we knew C sucked in the 90s, too, but we had three decades of over-confident dudes saying that was user error and _they_ were smart enough to use C correctly.

Then Rust came along and those dudes either tried to say it was too hard (odd contrast, that) or it’d be too much work to rewrite things.

I’m hoping that LLMs will lower the cost of porting for enough people but what would really help is shedding the idea that the status quo is sustainable. Having consequences for sticking with C would help but we need a way to get companies to pay to migrate without destroying open source.

reply
It's also valid to say that Rust's memory safety is good but the rest of the language sucks. Just as annoying as the over confident c-heads is the rust community that made even countenancing working on an alternative memory safe system a moral sin, which stifled innovation in memory safety. Hell, people barely know that Ada/spark is memory safe (or that sel4 is memory safe C)
reply
Aside from the overhead of borrow checking, what about the rest of the language sucks, in your view?

To me, its type system and traits design make it one of the nicest mainstream languages. I'd rather be using a functional language, like Haskell or an ML, and I do sometimes, but I don't pick my work based on that. I work regularly with Java, Go, Typescript, and Python, and out of those I find Rust significantly superior on several dimensions.

Python is terrible, Go is basically weakly-typed Python, Typescript barely deserves comment other than that it's impressive that such a fancy type system was bolted onto Javascript, and Java is basically COBOL at this point - reliable, but clunky and showing its age. Attempts like Kotlin to leverage the JVM aren't very convincing.

Gleam looks interesting but I haven't had a chance to work with it yet, and it's hardly mainstream.

You mentioned Ada/SPARK, but SPARK is not really for general purpose programming, and Ada in general has a similar issue to Java, but worse because of its age. It's as if someone really liked PL/I but decided it needed more features and a syntax closer to COBOL.

reply
lack of a stable ABI is a big one
reply
> what about the rest of the language sucks, in your view

I will chime in here to say that the adoption of async was imho a terrible decision and led me to abandon Rust for most purposes. I want proper threads and CSP-based abstraction.

reply
Ownership model is perfect for multithreading but then Rust goes all in on async like WTF?
reply
What overhead from borrow checking?
reply
I imagine they are referring to the cognitive overhead of learning to work with it.

Whether that is “something natural to you”, “something you learn and adapt to” or “something you fight with forever” seems to vary per-person, with I suspect, most people falling into the forest 2 categories.

reply
This really isn't true. It's true that those people saying those things existed, but much more recently. Back when all the C/C++ legacy code we depend on today was being written it was the performance penalty imposed by safety that forced developers down that path, not some lack of interest in other options. We were also transitioning from a world where a fair bit of systems code was written in even lower level languages than C (assembler, BLISS, BCPL). The world we're in today effectively began around 1975 when people figured out how to make a C compiler run on an 11/45. All the decisions were pretty much locked in from there on.
reply
What a strange takeaway. This is exactly how a server at a restaurant thinks about AI. Just the shortest term goals of keeping the status quo. They think that if they bury their head in the sand, everything will continue as normal.
reply