Given that in this case the workaround is as simple as disabling http2, there is honestly no reason why F5/Nginx or any of the reporting teams couldn't at least put out a PSA to disable http2 for the time being before basically telling the whole wide world this exists and how to exploit it.
Especially for software like Nginx which powers a huge part of the internet.
You basically get to piggyback off of a large chunk of big tech's infrastructure and security work. Obviously everything has the risk of security flaws (and I kind of hate that default Envoy builds ship with half-baked filter impls) but it's a generally more mature ecosystem than projects with a lot of smaller/legacy users like nginx
The main reason I see to not use Envoy is that it's very annoying to set up and build, and requires familiarity with things like grpc/proto/low level networking to get right. But if you're using grpc and http2 I think you're generally past that "infra overhead" threshold already and it's going to better serve your needs.
I have pretty much only worked at big tech or my own infrastructure startup, and did some systems/infra work at Google using Envoy for a while. I can't say it's 100% better than nginx but I can say that I've evaluated nginx and a lot of software like it (ie with a similar customer base, ecosystem, and history; like coturn) over the years for my personal use, and every time I've immediately been annoyed by constant security issues or poor http2/L4 networking support that I don't see when I use what Netflix/Google/etc use.
The documentation, DX/UX, and discoverability for these features are pretty bad. But they work well if you know how to do it.
Implementing a custom HTTP filter itself is actually easy, I can share examples I did in the last year for a voice inference stack if anybody is interested. The problem is just that the devloop is very painful, because Envoy doesn't make it easy to know why something isn't working (this is actually most of what I did with it at Google, tons of testing/verification and compatability infra for relatively simple http stuff) and the builds take a very long time.
Envoy is only attractive at a higher scale.
In a multiple node/distributed architecture you need away to coordinate rate limits across nodes and not try to fight the CAP theorem.
There is a way to do this [2] [3] and although it sounds complicated the shape of the solution (delegating to a third party service that aggregates and provides client-level rate limit metrics across multiple proxies) would still be the same any way you cut it.
[0] https://www.envoyproxy.io/docs/envoy/v1.15.2/configuration/l...
[1] https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/extensio...
[2] https://www.envoyproxy.io/docs/envoy/v1.15.2/intro/arch_over...
[3] https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/config/r...
I use JWT-based sessions + a remote JWKS (which is remote static content that Envoy can cache) + either a third-party IDP or dedicated login service (this being the only application logic besides Envoy's routing exposed to the public Internet). That way, every envoy auth check is immediate and stateless.
In this setup you don't really need a WAF because all unauthenticated requests redirect to signin or 403, which get strict rate limits on login. An abusive peer that sends a ton of traffic anyway gets blocked at the network/firewall level outside of Envoy. The tradeoff is you need to decide that you aren't going to do unauthenticated SSR, and I will admit this pretty complicated to set up in its entirety.
Should your distributor fall of the face of the Earth and you still need to patch software, it is often the fastest way to take the source package, put the security patch in the patch directory, and run.
There is more it, such as versioning it between the last release but below the official upgrade so it's convenient when the offical one comes along, and preparing keys if you require signatures, but that's basically it.
You would have had to enable http2 and have a buggy Linux kernel to be vulnerable.
If I wanted to build my own software from source every time I'd be running Solaris like a virile man from 2004.
stable-alpine-slim docker iamge is only 14MB?
Which may be good or bad.
Ubuntu and other distros have not kept up.
Podman container auto-updates to the latest upstream. It is a minimal surface area given alpine images. Security first containment.
In this day and age with AI attackers and zero-days it is absolutely daft to still treat servers as the old-school pets model.
Doesn't this make the system more susceptible to supply chain attacks?
Um, what does this have to do with NGINX?
Did you talk to it like a pirate?
And here lies the risk of AI agents in cybersecurity: the speed of processing for monitoring, scanning, identifying security fixes in open source, writing exploits for affected versions, and attempting infiltration, is a matter of minutes now.
It is not "by anyone smart enough". It is by "anyone ill intentioned enough". They don't need to be super smart anymore.
There’s no good solution, it’s a minefield and the only choices are what you’re willing to lose to maybe make it out of it
Cant win.
Seriously reconsidering if open source has a future nowadays
The issue is that we get dozens of PRs, generating a bunch of code is trivial but reviewing and merging it and making sure it doesn’t break in interesting ways is still pretty hard.
Personally I lean towards not accepting code contributions anymore, but accepting design documents for features and l bug reports etc
It's not a perfect solution of course, it just takes OSS back to the fairly broken state it was pre-AI. Still, it might be an improvement and should be at least considered.
I’m not sure how we could enforce an 0 LLM policy in practice though…
*In C or C++.
C and C++ will stay around for a very long time, even with AI powered systems.
If anything, maybe that is a call that even if implementing compiler backends is hard and time consuming, bootstraping compilers, and new approaches to OS design, are the only path away from such dependencies.
Legacy 32 bit users with NetBSD and Hyperbola can survive a little with Go based tools/small browsers/mail/usenet/IRC clients and some web browsers with Bubblewrap, but not forever.
https://vulkan.org/events/vulkanised-2027#program
The only thing closer to it is WSGL.
Also there are much more operating systems than those examples.
Security is hard and is mostly about the number of engineers and amount of time spent looking at a specific implementation. When writing code, do you think about the problems of leaking information due to different operations taking specific amounts of time? That's an actual concern when writing a prod service that uses the network. And it has to be balanced against scalability and issues caused by DOSes. It isn't trivial and it isn't binary...there are always trade-offs. Nothing will ever be perfectly secure and usable and scalable.
But doing that requires collecting the whole chain. In the current environment we might be finding and patching these smaller bugs before they can be put in to a chain. As long as you are keeping on top of updates..
Well, actually four: https://lostartpress.com/products/the-stick-chair-book
According to the author, Troy could do the work of three people. That would make him a 3x woodworker, I guess. And he was facing automation in the woodworking industry.
"But Troy decided to compete with the machines on their own turf. He tried to become a one-man factory. He succeeded in that. But the most efficient one-man factory cannot compete with a factory driven by a hundred men.
So no matter what Troy did, the prices for work (good or otherwise) declined. Ikea’s well-designed kitchens and Value City’s cheap junk made it impossible to survive and live well.
So Troy struggled with his business. And that’s nuts. He should have had more business than he could handle. But like many professional cabinetmakers I know, it became harder and harder for him to make ends meet every year. His cabinetmaking business is closed, and the last time we talked, Troy was working for his wife’s successful tax business.
Who is to blame for this? We are.
We have become a culture that is obsessed with price more than any other attribute of the things we buy. It doesn’t matter if the item is ugly, poorly made or constructed of materials that cannot be recycled. All that matters is if the price is low enough.
Because the price of our household objects has hit rock bottom, if an item breaks or starts to look dated, we can throw it away and buy some- thing else. For the first time in human history, manufactured furniture is shockingly inexpensive.
So it’s no wonder that artisans are exiting the craft. It’s difficult to compete against furniture that costs less than what you pay for your raw materials.
Lesson: People cannot imitate machines and defeat them."
Appreciating good design, quality craftsmanship, having taste in food.. having these "elevated" tastes in multiple areas is very rare for humans. Take a look at a musician - they can tell you precisely what makes a quality violin, but they might not care that they eat microwave frozen dinners. You compound that across the population and the artisans will always lose out to the mass-produced slop because it's impossible to naturally get a critical mass of people to care about that quality curve. You have to sell them on it.
Troy struggled because he failed at marketing, not production.
I think I've learned that most people don't want to put in any effort to learn or appreciate things that may be of nicer quality. Those Japanese knives didn't do any good for them because they didn't spend time learning how to use them properly. In fact one of them used them improperly and ended up needing stitches. So in the end the crappier version is no different than the nicer version, because crappy and nice are about the potential of the item, and most people don't see potential. Does this Ikea dresser hold clothes? Great, who cares that the other one is of nicer wood, the joinery is done by hand, and it'll last multiple generations.
I see a lot of interest in woodworking on HN. Is that because Jesus was a carpenter and HN folks are interested in woodwork because of the larger context and history shaped by Christianity? What is your motivation for woodwork? Is that because we see a concrete finished project instead of vague intellectual stuff which has no seeable boundaries?
The comment you replied to is shortcutting the joke, assuming the common riff of becoming a carpenter is well known, and jumping ahead to asking about tutorials for woodworking as part of that process.
Preposterous, even.
That said, my RHCE CISSP self is going to go outside and spend some time with my flock of chickens and guinea fowl.
While Claude writes Ansible to update the fleet to use nginx's deb repo.
The account is from 2017. There are enough comments before LLMs were a thing. Given his karma score, unlikely to be a bot.
> Hey so, does anyone know of a good tutorial on getting in to woodwork?
Some people really don't get these type of things.
https://x.com/Laughing_Mantis/status/2123146699396858260
Its a non issue, but people just want to be told AI is God or something and the world is ending.
Its a non issue.
Are we going to discover in a year that every server online this month had a rootkit installed? There’s got to be thousands who are too small time to wipe and reinstall.
Worse patching adds instability into a system, especially when you don't have much time to test.
Also in the past attackers were willing to sit on exploits longer and not do things with them, now they have to act fast before someone else finds it, leading to more events faster wearing your security and patching teams out, and making users life suck.
Things are going to get much worse before they get better.
I guess you could make an argument based on some estimate of how fast exploitable bugs are newly introduced. That also seems dubious.
Survival of the fittest I'm afraid will build security here. The teams that barely ever patch or don't have "time" for maintenance are going to get burned. The "we haven't updated in 7 years so things are stable" teams can't survive.
Collectively, we owed about 40+ years of tech debt on non-secure code (e.g. automotive industry, industrial environments) that will be torched in the next few years. Luckily, AI will generate code faster enough to make that plausible.
That's not happening at nearly the same pace, both for assymetry between what it takes to break something and what it takes to fix it, but also because that fix must navigate bureaucracies. Also, the leaders need to actually care, both of which are hurdles. Bureaucratic navigation and consensus finding has not sped up at all with AI.
Compare that to independent script kiddies with $600/mo Claude subscriptions able to find 0-days in an hour and write a worm to exploit it in four.
> AI is securing the internet.
No doubt there are points of view where either one, or the other statement, or both, or neither, are true. You can really go full Schrodinger and put a probability distribution on everything.
But in this day and age, whatever is happening, the better question is:
Is it good for me and, beyond me, for the world in general?
it was previously expensive to attack, so people spent less money on defense.
It's cheaper to attack, and so people will increase their budget on defense. But it's a lagging cause & effect situation
Even Kimi K3 was only shipped as a lobotomized version of it.
A huge fraction of devices and servers are on old versions, not actively updated or managed in any way.
If you're not going to take care of it you've created a public nuisance.
There are two internets. One of them is open, less-commercial, and largely built and maintained by people whose first job is not the work they do to maintain this internet. That group includes a ton of open source software, millions of small servers scattered around the world (virtually private or otherwise), and is generally best-effort.
The other internet is fully commercialized, often ad-driven, fully integral to the data broker economy, and centralized behind a few companies that pay as few people as possible to satisfy the occasional PR statement that, "we take security seriously and...".
The next stable equilibrium here is not "we get a secure and open internet". The next stable equilibrium here is, "we have no open internet, and the internet is now managed by just a few companies."
We're already halfway there. Google and Microsoft combined already account for somewhere around 30% to 40% of email traffic. 30% of all websites are now behind Cloudflare. A few companies dominate social media traffic. Amazon and Shopify combined are half of US e-commerce.
Every single short-response-window exploit like this just pushes a few more percentage points of traffic into a handful of companies. Cheer for that, I guess.
When I have time to daydream, it's increasingly about building a new, underground internet, and how best to disadvantage bots and BigCo conglomeration from the start.
The real problems here are that we did a crappy job in the first place and left a huge surface area of risk with the blind assumption that we didn't do a crappy job.
I am hoping to see a change in attitude to risk and quality in the industry but I get the feeling that's asking for too much.
Organized Cybercrime?
Digimafia?
e2p3yHizAungrCeKum7ifrssu32lnzEnyRoGdcAo23m2dNapvoGmymid.onion?
A more apt analogy would be that pretty much all consumer-grade locks are very easy to pick, despite being marketed as secure. I think in this case the lock makers should be blamed, not the lock pickers.
The lock makers are us - the IT field at large. As individuals we might not be to blame, but we are still enthusiastic parts of the greater movement that's issued vast digital complexity to the world without truly understanding it.
Now we are starting to understand it more.
Please stop willing things into existence.
Obviously, AI makes these things easier to find and exploit, but this isn't even close to the first time that 'thousands' of hobbyist (or small business or just not very competent) server admins are going to get rootkitted en masse. I worked at medium-sized infra-as-a-service company when the above was discovered and there were definitely thousands of compromised machines just there.
This is the sys/devops time to shine. Everyone needs to have an upgrade and an emergency plan, and everyone who was previously only doing conventional codemonkey needs to start learning more of the stack to join in on that.
I do think the world is going a little insane tho. People suddenly believe in security through obscurity because, uh, AI bad?
There might also be a day in the future where our current encryption methods are broken - it doesn't mean they're insecure and vulnerable today.
We have a billon hacking experts now is the big change.
Care to elaborate why you think that sufficiently paid humans would not be able to find the flaws AI now finds, for cheap?
Harnesses allow LLMs to build & process large stores of data to reuse later. Imagine an agent that spends 3 days breaking into company A, records what it learned during that process, letting it break into company B in just 3 hours.
Defenders win by having the best security 24/7 and must trust that other services they depend on are doing the same.
An attacker wins when they can breach any defender's security.
I'm not making a Rust advocacy argument. I'm just saying: it's true that integer handling issues are endemic to mainstream programming languages, but the integer handling itself is almost never the actual sharp end of the vulnerability.
Integer handling mistake -> memory corruption -> RCE.
That's the pattern. In a memory-safe language, the integer handling error has no link to memory corruption (though it will to other logic bugs!).
Why would CPU designers make a trap if languages do not require it?
"Go fast and get your things broken into"
The circle of life.
I'd be more worried if the response isn't more secure development practices in better languages, but instead use of AI as a defense. Then we're truly talking about burning down the internet.
A dead box isn't part of any bot army.
Tech that becomes infrastructure like web servers doesn't move so fast and it shouldn't. Most people and companies should be relying on relatively tried and true software, so this phase of AI finding vulnerabilities in it should pass and things could get better.
Like Java. Surely anyone can't mess up simple things like base64 decoding in it...
That's orthogonal to the point I'm making which is that no amount of reviews is going to make every single bug go away. Because, once again, humans and AI are fallible.
Sure, there are niche things. You can’t use US ciphers in China and vice versa.
But overall, one day, the intelligence communities world wide just decided to not raise a fuss about everyone having solid cryptography.
It makes me think if we did find out that every server out there had a route kit… it would make that part make sense.
All bugs are shallow given enough eyes, and the marginal cost of eyes is now ridiculously low.
Imagine a alternate reality where Ada replaced C by 1995. We would be in a MUCH better place.
Then Rust came along and those dudes either tried to say it was too hard (odd contrast, that) or it’d be too much work to rewrite things.
I’m hoping that LLMs will lower the cost of porting for enough people but what would really help is shedding the idea that the status quo is sustainable. Having consequences for sticking with C would help but we need a way to get companies to pay to migrate without destroying open source.
To me, its type system and traits design make it one of the nicest mainstream languages. I'd rather be using a functional language, like Haskell or an ML, and I do sometimes, but I don't pick my work based on that. I work regularly with Java, Go, Typescript, and Python, and out of those I find Rust significantly superior on several dimensions.
Python is terrible, Go is basically weakly-typed Python, Typescript barely deserves comment other than that it's impressive that such a fancy type system was bolted onto Javascript, and Java is basically COBOL at this point - reliable, but clunky and showing its age. Attempts like Kotlin to leverage the JVM aren't very convincing.
Gleam looks interesting but I haven't had a chance to work with it yet, and it's hardly mainstream.
You mentioned Ada/SPARK, but SPARK is not really for general purpose programming, and Ada in general has a similar issue to Java, but worse because of its age. It's as if someone really liked PL/I but decided it needed more features and a syntax closer to COBOL.
I will chime in here to say that the adoption of async was imho a terrible decision and led me to abandon Rust for most purposes. I want proper threads and CSP-based abstraction.
Whether that is “something natural to you”, “something you learn and adapt to” or “something you fight with forever” seems to vary per-person, with I suspect, most people falling into the forest 2 categories.
echo 'install sctp /bin/false' > /etc/modprobe.d/no-sctp.conf
echo 'install dccp /bin/false' > /etc/modprobe.d/no-dccp.conf
echo 'install rds /bin/false' > /etc/modprobe.d/no-rds.confhttps://xint.io/blog/nginx2root-remote-code-execution#23-com...
Workarounds if you're vulnerable and can't update immediately:
https://xint.io/blog/nginx2root-remote-code-execution#33-eme...
But when newer software has the same vulnerability, you do not see the same kind of comments.
There always seems to be a latent stance in IT that newer is better.
This isn't true! Just look at this thread, people suggest replacing nginx with newer but less widely used and less battle-tested software, as if they're superior alternatives or don't have RCEs in their own history: Envoy, caddy, traeffik, Qbix.
Go isn't quite a memory safe language, but it provides fewer footguns.
On the other hand, Caddy has a lot more features, so there's more code that can contain exploits.
We have evidence that large unsafe codebases suffer from more and more severe CVEs than those who are not in memory unsafe languages.
In other fields of engineering, people use stats and actually measure things like this, and use the results to determine what to do. They don't just assume something is better because its "battle tested" (whatever that even means?).
Plus, Caddy itself wasn't vulnerable in this specific case. There wasn't much the Caddy folks could've done to protect against a problem in the Linux kernel itself.
All the remaining C code needs to die in a matter of months not years.
Why so? are most of these vulnerabilities from C's memory model?
the certificate for it was issued on the 2nd of November
if you were scanning CT logs you would have had a 16 day head start on RE'ing this
there are more examples in the current CT logs
https://api.certspotter.com/v1/issuances?domain=nginx2root.c...
ping2root•com is available. :)
If you want to promote responsible anything, go for responsible writing of software in the first place.
Python simply can't overflow, since its integral type is not a fixed width.
But it is the sort of correctness thing that I do wish languages would pay more attention to.
Python master race
Is this some kind of record for bug collision?
If anyone gets a hint of the space a vulnerability occupies, just point your agent at it and you have a pretty good chance of finding something. (See java4shell and the NYT reporter who was able to replicate the problem with $200 of tokens)
But it's like inventing a hammer and finding you can smash whatever you want with it.
Soon you'll be building houses with it.
Maybe that's being glib but it's just a tool being applied to a world that is still largely prestine from its impact (pun unintended).
.."Soon some of you will soon build houses with it, others will be building rocket-powered precision aimed intercontinental hammers, to smash whatever you want."
i.e fast followers of the researchers who don’t have the resources to burn through checking all of a tool.
What are the odds several humans independently uncovered a complex vulnerability like this simultaneously?
It seems most logical for trusted parties to scan the frontier before passing those capabilities onto anyone with an account, and it seems like that is exactly what they are doing.
We are in an extremely unique period of time where the cyber good guys can have machine guns and cruise missiles while the masses only have sling shots and catapults.
Consider a world in which corporations like the Ubuntu Foundation or Microsoft experience zero public pressure to fix such vulnerabilities but instead fix it only when a major customer notices their servers getting hit.
Imagine if instead of "you've got 30/60/90 days to fix this thing I noticed before I post it publically" it was "okay, well, take your time I guess, I just hope North Korea doesn't notice the same issue I noticed, which is especially likely because they have far more resources than I or my small team does to point at discovering this kind of thing, while you're dragging your feet fixing it."
The reason people want shorter disclosure windows is because the shorter the window is the faster things get fixed for the average users and smaller companies.
Massive monetary loss, explosions, and dramatic failures will push computer security in a year where it should have been decades ago.
> Not yet. We are not publishing the exploit now because the risk is too high.
So who's the judge of where to draw the line between exploits that are relevant to publish, and which are "spam". You?
Or do you just think that companies shouldn't be allowed to show the work they're doing for marketing?
Certainly a stressful time for sysadmins everywhere, let us hope your coworkers are picking up your beer tab
At the rate of development, also, will AI developed in 6 months from now be able to break a system that was secured by todays AI?
Now, you could argue that writing secure code ex-ante is harder than discovering an exploit. But you can hypothetically run the same "exploit finding" process on a first pass of code before it ships (e.g on the PR) given low enough token costs, which is where we're headed.
Even if that's true, one should not forget that:
When the hackers have breached something, awaits a reward.
E.g. a Bitcoin wallet, or learning something about a political competitor and getting rid of them.
(So the hackers can spend lots of ...)
I wonder about that though: if the price of the squeeze collapses to 0 because of this (think of what models and prices might be like a year from now), I think I know where all those out of work mathematicians can find gigs. We are going to need to invent a layer on top of, an evolution of or some new concept a level above "computer programming" that deals with meta-organization of code.
I got worried for a sec, then remembered all my nginx lives behind HaProxy and it looks like nginx is only vulnerable if its going http2/TLS directly.
Now I sit and wait for an HaProxy announcement...
ps. if you are on FreeBSD and doing TLS directly
sed -i.bak -E 's/^([[:space:]]server[[:space:]]\{)$/\1\ http2 off;/' /usr/local/etc/nginx/nginx.conf ;service nginx restart
I know times are changing, it's just curious that nobody seemed to care until now.
A lot of stuff that I avoided for years I finally feel like I really need to do now but it also has never been easier to do.
My box got hammered daily with port scans etc before.
Fortunately with Tailscale making it super easy to keep stuff out of the open internet, I don't have much, but even then I'm thinking of taking what I do have down. We see these big CVEs coming across, but I guarantee there's a large number of 0 days sitting around in peoples' stashes all over the world that we don't see.
Debian is unfixed too
https://ubuntu.com/security/CVE-2026-123633
https://ubuntu.com/security/CVE-2026-123007
Neither has Debian:
I wouldn't expect any distro maintainers to react in hours. Distros are too complex to dump updates in rush.
Start to use Docker/Podman/systemd-nspawn.. and isolate public services to necessary needs (static binaries, drop capabilities, default drop rule firewall between containers, r/o storage, access logs bind mounted as rw file, memory temp dir with noexec). Container images are patched the quickest.
Just keep that protocol disabled for now. It's apparently too difficult for any human to write a secure implementation of http2. (As for http3, forget about it, we need AGI for that.)
edit noted it's about a Kubernetes controller
New code means new bugs, and losing the many years of accumulated bug fixes and hardening present in many critical pieces of our infra by rewriting in rust is not a desirable tradeoff. The new world we live in where fv has become many orders of magnitude cheaper also makes rust seem much less attractive.
While generally more of a hassle than Fil-C, the scpptool approach provides an incremental path to optimal performance (for performance-sensitive sections of the code) via migration to a (statically-enforced) memory-safe subset of C++. And unlike Fil-C, the scpptool solution essentially maintains the portability of the original code, as well as its deterministic performance and resource usage (i.e. no GC). It also supports ABI compatibility via "unsafe"/"check-suppression" annotations. The scpptool approach is not so much an alternative to Fil-C, as it is an optional path forward for C (or C++) code bases that may (or may not) be already using Fil-C for memory safety.
[1] https://github.com/duneroadrunner/SaferCPlusPlus-AutoTransla...
It's time for performance above everything else to die, especially on anything and everything that gets anywhere close to untrusted data.
The project isn't currently open source and there's next to nothing on the site to reassure you who's behind the server gateway you'd be running.
Googling the company in the footer produces this, but it also doesn't have much: https://www.signeen.com
And parts of that involved specifically replacing Nginx in their architecture, so it's hardly irrelevant here.
https://blog.cloudflare.com/20-percent-internet-upgrade/
https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...
> With those assurances we can focus more on how a change to our service will interact with other services or a customer's origin. We can develop features at a higher cadence and not be burdened by memory safety and hard to diagnose crashes.
> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception we’ve served a few hundred trillion requests and have yet to crash due to our service code.
> In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible.
is this just the continuation of the previous point (compiler/borrow checker can have bugs), a nod that you can import unsafe dependencies, or is there a soundness gap in rust that I'm not aware of?
So I'd say it is about a more holistic risk-analysis not just handwaving "all code can have bugs, duh!"
That is a far smaller surface as seen by research into incorporating rust in Android leading to none of the high profile vulnerabilities C and C++ led to previously.
On the Android point, I remember those metrics were highly flawed and didn't account for the inequality of substructural type ergonomics in different contexts. You can get very nice metrics by rewriting things that fit to the grain of safe Rust, which gives you an inaccurate picture of how much unsafe code is "real world". LLMs have changed the field here, but it's worth pointing out that blog post you're referencing was more self-congratulatory marketing than serious science.
Also problematically, I remember it being highly misleading in that the actual observed result was lower severity in vulnerabilities, but a very minimal effect on the total number over all. That's certainly significant, but it wasn't how it presented itself, which is rather problematic if you expect people to take it seriously as objective research.
Measurements probably real, but not worth citing. It's not like on their own blog they're under usual academic journal pressures to overclaim results, p-hack, and engage in other misleading behaviors.
https://blog.google/security/rust-in-android-move-fast-fix-t...
In theory, that could be possible. In practice, there have been many many case studies demonstrating the opposite and not that many case studies demonstrating in favor of your argument.
e.g. Cloudflare replaced Nginx with Pingora: https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...
> With those assurances we can focus more on how a change to our service will interact with other services or a customer's origin. We can develop features at a higher cadence and not be burdened by memory safety and hard to diagnose crashes.
> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception we’ve served a few hundred trillion requests and have yet to crash due to our service code.
> In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible.
The CVE counts for new parts of the Linux kernel written in Rust are much lower than the (recent) CVE counts for even mature, years old parts of the Linux kernel written in C, to say nothing of the historical record.
Additionally, check what the inventors themselves of UNIX and C did after being done with them, and what they did afterwards in terms of operating systems and programming languages.
Yet, some folks are more attached to C than the inventors themselves.
Ferron seems much better, but I'm also not convinced that it's really production ready
If e.g. the standard array type in a language has built-in bounds checks that error out when you try to write to an index that's outside of the array, you can't trivially create buffer overflows simply by writing to the wrong index or by trying to write n + 10 bytes into an n-byte array.
But C in itself has no guardrails against something like
char buffer[100];
strcpy(buffer, some_user_supplied_string_with_200_chars);
Of course most memory safety bugs aren't trivial like that since C programmers have hopefully learned to avoid the simple mistakes. But there can be more complicated and difficult to spot cases, and C has no guardrails against those either.Practically impossible? Yes.
SQLite is probably the most thoroughly tested and carefully developed C library in history and they still found a handful of memory safety bugs every year.
The name refers to the number of days since public release. "Exploitable 0 days since release". If you found a bug before release, then that's... Just a bug picked it up in normal development lifecycle.
I think that's the fundamental disconnect in the conversation here. You think I'm talking about the dev because it's not released yet but just because it isn't tagged and released doesn't mean it's not done in the open where hackers can see it and their AI can scan it.
Caddy, Traefik, or even Envoy may be better choices.
I am running the official nginx docker image by itself in front of all my services, which are their own docker images. So it's easy for me to update just one of them. This works for me because I want my dedicated server to run all my projects with nginx in front. If I were running Discourse I would want it to not bundle nginx because I've already got one of my own. So that's interesting that they do that.
Unlike IPv6 it doesn't need any OS or ISP support, just a browser that's newer than 2020. Everyone outside of air gap networks and government sites should be switching.
https://portswigger.net/research/http-desync-attacks-request...
Unfortunately, there are still plenty of customers behind CDNs which don't support HTTP2 for backend origin connections.
There are many legitimate use cases for h2 (eg bidi streaming, latency improvements) and h3 (eg seamless network switching).
If you don’t need these features and just serve vanilla text webpages that’s also fine to just use h1.1. Blanket statements that h1.1 is better is just, wrong. Context is important on choosing what’s best for your environment.
Which, to be fair, is most sites.
Simple is good.
Because clearly the patching treadmill is not sustainable, you will need a a second line of defense and reduce the blast radius of every individual vulnerability.
One, as the RCEs are coming out, admins are on edge and paying more attention, so more likely to catch the annoucement on their platform. "Oh, that's on PHP, we don't use this. Oh, wait, another one?"
Two, all of these hitting at once, while they may expand the surface for attackers, it also spreads their attention. They only have so much time and bandwidth. When a new shiny RCE hits, they need to consider "do we continue to try an prosecute the old one or jump on this bandwagon".
So, in the sense, it can actually slow down the actual potential impact of the revelations.
We can only hope this gives users time to get their systems up to date before they come knocking.
I mean, I know its a reach, but...who knows.
Only a little bit. An attacker will just hand this to an LLM with a "ping me when this is ready to use..." prompt.
Seriously though, with how CVE disclosures are handled these days patching is a race against the clock
> Error: API Error: Opus 5.5's safeguards flagged this session
Literally the first message, lol.
It'll take a few years for the market to adjust. If nothing else, businesses are good at figuring out ROIs, eventually.
Only half joking.
Cue xkcd
EDIT: Okay, this comment was snarky and not in good spirit. To be serious, I think we are increasingly seeing that the notion that humans can write non-trivial amounts of C correctly is false. Especially in the face of AI-assisted vulnearbility hunting these programs a crumbling. This is not an indictment of humans, it's a sign that C is not equipped to meet this moment.
https://github.com/nginx/nginx/commit/e27f5fa84fe6e98505e70b...
Given that on modern CPUs, cycles are quite fast, and memory latency is quite high, always-checked integer operations sound like an extremely good idea.
C is like a Winchester Model 1897, not saying that it is a bad gun, it is what we call it the good ol' Trench gun, as it ran over the war tunnels of the Great War, and it still packs a punch today, but it only has a half-cock for some level of safety, and there is no trigger disconnector either, so you can easily slam fire it. So it is way too easy to shoot yourself in the foot.
Nowadays we like going for a AKM/AK74, for something maybe more complicated but far more reliable at the same time. That's why we want Rust.
Otoh I'm not sure what http2 is good for (I've asked around today, some people said the multiplexing speeds things up on high-latency connections. I'm still working on a realistic benchmark since I couldn't find any but it's not looking great so far) and it's default disabled so comparatively few sites are affected. Getting rid of it is easier than waiting who knows how long for it to get packaged
There are also features on top of http2 that a downgrade may break: gRPC and Server Side Events for example. The later is compatible with HTTP1.1, but could fail due to connection limits, as multiplexing isn't only about speed. Some web apps may handle that so gracefully, it turns into a performance thing again, but others could halt and catch fire.
That's to say that this is our new normal. Something that would have taken months or years to figure out can now take an AI hours if not minutes if you have enough compute.
https://security-tracker.debian.org/tracker/CVE-2026-123007
Debian still vulnerable in all versions.
12 years on, OpenBSD's port of nginx still maintains its default chroot(2) patch.
https://raw.githubusercontent.com/rnagy/nginx_chroot_patch/m...
I recommend reading Reyk Flöter's AsianBSDcon 2015 paper about nginx and the thing that replaced it in base.
It is completely realistic that we start seeing a lot of bugs a human can’t (reasonably) exploit, but AI can.
I mean sure if we close our eyes and ears we can ignore the elephant in the room that is proprietary, closed source software.
HTTP/2 again. First Apache, now NGINX
I rarely use HTTP/2, I rarely use a popular web browser, I use a TLS forward proxy that has HTTP/2 support but I prefer HTTP/1.1
I find that 1.1 is faster for the way I use the www (text-only, no images, CSS, Javascript, cookies, advertising, tracking), plus the 1.1 protocol comes from academia not Google, Akamai, etc.
Edit: All good. They released the new versions within 30 mins.
[1] https://launchpad.net/ubuntu/+source/nginx [2] https://tracker.debian.org/pkg/nginx
Given how common it's become to constantly scan popular repos for n-days, I expect someone will have already picked it up and started work on an exploit. The workaround for this (disabling HTTP/2) is also comparatively easy to what we had with PHP2Shell just a few weeks back, so I think it makes sense to release this.
We have people doing this on my side project and then reporting the fixed issues to try and get CVE 'credit'. Tiring.
Perhaps it was already being exploited?
We've literally decades of evidence that we cannot write software securely in languages that aren't memory safe, for all sorts of reasons (and it's wrong to look at it as a developer competence issue, even though sometimes that's going to be the case, even extremely experienced and highly competent engineers write memory safety bugs in critical code we rely on day in day out.). Not leaving yourself open to such problems is an entirely logical conclusion. We know a large percentage of vulnerabilities in software are caused by it[1]
Pick a language that doesn't have these risk factors, or pick an enforceable subset of one that makes such impossible.
"Insanity is doing the same thing over and over and expecting different results."
[1] https://www.chromium.org/Home/chromium-security/memory-safet... 70% of Chrome vulnerabilities caused by it. Similar stories for Microsoft products https://github.com/microsoft/MSRC-Security-Research/blob/mas... and Mozilla: https://hacks.mozilla.org/2019/02/rewriting-a-browser-compon.... Curl is at 40%, and dropping because Daniel Stenberg has provided various helpers that avoid it (and I believe they've now migrated almost everything over to them) https://daniel.haxx.se/blog/2023/12/13/making-it-harder-to-d..., so there are definitely ways to do it, we just... don't.
Today we are going to take a look at how to avoid this vuln by not having Nginx and not using HTTP/2 (for most use cases).
We will assume that you are using Nginx for any of the following: Serving files or serving dynamic website content through some integration with a programming language.
We will be using python, but you can use any programming language that has a native integration with sockets through it's C interface (literally every language:
Step 1: import socket # import the socket library from glibc or glibc.
Step 2: socket.socket() # socket.socket()
Step 3: socket.bind((PUBLIC_ADDR,80)) # bind before listening obviously. Use a local address if you are below the IPoverty4 line.
Step 4: conn,addr = socket.accept() # this creates a connection, wrap it in a While True and a try catch for production. Sprinkle in some import threading to support multiple concurrent requests.
Step 4: req = conn.recv(4096) # split on b"\r\n\r\n" to separate headers from body, read HTTP RFCs for the rest. Feel free to reject HTTP/2 requests, they'll probably just resend as HTTP/1.1
Step 5: #Fill in the details, most likely if you are using nginx, you will end up connecting two sockets with pipe, or just storing and forwarding to another tcp endpoint. Again take a look at the HTTP RFCs and the glibc socket and stdio manuals.
Thank you for reading my tutorial, and best of lucks.
This webserver is not vulnerable to PHP2Shell nor this TLS issue, because it's completely userland PHP:
https://github.com/Qbix/webserver
It's now been battle-tested in production, and also supports websockets, etc.
(However, it doesn't support sendfile, because PHP doesn't have support for that, so it's not as fast as nginx for sending static files. This shouldn't make a difference, though, if you use a CDN in front.)
In fact in general there were waaay less CVEs for Caddy lately compared to nginx
I guess it was the right move.
Some of these things could've been coincidence but I remember seeing an apache RCE a few days back and people said, oh imagine using Apache, I use Nginx and now not a few days later Nginx has RCE.
And this is just a very very small iceberg of all the recent RCE cases with a serious uptick that I have just observed on Hackernews alone.
Basically, there is a sort of justification that people provide with one RCE is that that software X has had issues and to use Y software, but now that X,Y software and A1...Z1 all are being found to have quite severe bugs. I can't in good faith just consider it to be a coincidence.
Though I do still hope that this might just be a coincidence and not a recurring pattern, because, A) I don't wish to live in that reality where each software that we trusted seems to have quite so many bugs and which would be exploited and B) Have it be continued for a prolonged period of time.
Though the plus point is that we are gonna find extremely resilient software after all of this hopefully as sort of a need in our times.
We might have to agree that the cybersecurity world seems to have changed (though I am no expert right now to comment on it but I am curious what experts think)
Theh people laughing at Anthropic and co for being cautios of Mythos just didn't understand whats happening.
Instead of finding, hiring and then finding security issues, you can throw money and compute at it. This alone is a complete game changer. Combined with models which are very very capable.
This can be potentially the death of certain programming lanugages or requirement of a complete rewrite. It can mean that only big companies have the resources and capacity to continue playing this game.
If you think you can do it or that anyone with AI dan be dangerous. Please, right now put everything down.. Go make a h1,bugcrowd acct and start making millions. If its as easy as having access to llms please go get rich. I assure you just having access to tokens doesnt get you very far.
I'm not saying domain knowledge is not required. But a lot of people do have domain knowledge and are prevented from using these tools. Even to secure own code bases. This sucks. A lot.
This is a monumentally dumb assertion that any Job Blow can just add a credit card and ram through these requests
So much cope on this site. Llms havent replaced the requirement of studying and becoming an expert.
But sure, keep telling yourself this is AI lmfao and anyone can hack because of LLMs.
Also, im not worried about anyone exploiting this. I have multiple nginx severs running http2. Im going to take a nap. After I wake up I MIGHT consider updating.
People in these comments who think they should have given people the opportunity to shutdown http2 in advanced, as if everyone with an llm is going to produce a PoC in 10 minutes. I got news for you, nobody cares about your server and nobody is figuring out how to use this PoC..
Im way more worried about Wiz, the literal Israeli malware/spyware company playing games than anyone else. Wiz is kind of lowkey evil and I wish Google hadnt of bought them.