upvote
Disabling http2 can be a big deal. My service is primarily made up of gRPC APIs. Not an option for me to disable http2. Luckily I'm not using nginx at the moment - but I easily could have been.
reply
If you're using grpc, http2, etc you're so much better off using Envoy proxy

You basically get to piggyback off of a large chunk of big tech's infrastructure and security work. Obviously everything has the risk of security flaws (and I kind of hate that default Envoy builds ship with half-baked filter impls) but it's a generally more mature ecosystem than projects with a lot of smaller/legacy users like nginx

reply
I'm not 100% informed on this, but it seems very unbelievable to me that Envoy could be considered more robust or widely used than nginx at any point in history including today. I started using ConnectRPC instead of GRPC almost entirely because it didn't leave me subject to (what I consider to be) an unnecessary dependency in Envoy.
reply
I guess I would just say, wordpress is also more widely used than many other projects, but that doesn't necessarily make it more robust or secure than the projects Google, Cloudflare, and security companies develop.

The main reason I see to not use Envoy is that it's very annoying to set up and build, and requires familiarity with things like grpc/proto/low level networking to get right. But if you're using grpc and http2 I think you're generally past that "infra overhead" threshold already and it's going to better serve your needs.

I have pretty much only worked at big tech or my own infrastructure startup, and did some systems/infra work at Google using Envoy for a while. I can't say it's 100% better than nginx but I can say that I've evaluated nginx and a lot of software like it (ie with a similar customer base, ecosystem, and history; like coturn) over the years for my personal use, and every time I've immediately been annoyed by constant security issues or poor http2/L4 networking support that I don't see when I use what Netflix/Google/etc use.

reply
Envoy probably has the advantage that it’s only a reverse proxy server, unlike nginx which has other uses too like a general-purpose web server with optional modules.
reply
You can implement custom http filters for envoy or use this for general web serving: https://www.envoyproxy.io/docs/envoy/v1.15.2/configuration/h...

The documentation, DX/UX, and discoverability for these features are pretty bad. But they work well if you know how to do it.

Implementing a custom HTTP filter itself is actually easy, I can share examples I did in the last year for a voice inference stack if anybody is interested. The problem is just that the devloop is very painful, because Envoy doesn't make it easy to know why something isn't working (this is actually most of what I did with it at Google, tons of testing/verification and compatability infra for relatively simple http stuff) and the builds take a very long time.

reply
On all my small projects (eg rails apps) the reverse proxy to the app servers is the same server expected to serve the static assets directly. In fact the majority of bot nonsense never goes through the proxy due to path rules.

Envoy is only attractive at a higher scale.

reply
Does Envoy support things like rate limiting?
reply
Yes [0]. For a single node you can just use this, it works like the Golang/language rate limiters you might already be familiar with [1]

In a multiple node/distributed architecture you need away to coordinate rate limits across nodes and not try to fight the CAP theorem.

There is a way to do this [2] [3] and although it sounds complicated the shape of the solution (delegating to a third party service that aggregates and provides client-level rate limit metrics across multiple proxies) would still be the same any way you cut it.

[0] https://www.envoyproxy.io/docs/envoy/v1.15.2/configuration/l...

[1] https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/extensio...

[2] https://www.envoyproxy.io/docs/envoy/v1.15.2/intro/arch_over...

[3] https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/config/r...

reply
I don't know what nginx security vetting looks like, but just being older isn't a security guarantee. I know Google spends a lot of manpower on making envoy secure.
reply
What WAF would you suggest to use with envoy? Have been working with Coraza and tbh satisfying is something else. So i wonder what someone that recommends envoy would recommend as WAF for it
reply
Personally I serve all static, public content off a CDN and only serve dynamic content through Envoy after it passes AuthN/AuthZ.

I use JWT-based sessions + a remote JWKS (which is remote static content that Envoy can cache) + either a third-party IDP or dedicated login service (this being the only application logic besides Envoy's routing exposed to the public Internet). That way, every envoy auth check is immediate and stateless.

In this setup you don't really need a WAF because all unauthenticated requests redirect to signin or 403, which get strict rate limits on login. An abusive peer that sends a ton of traffic anyway gets blocked at the network/firewall level outside of Envoy. The tradeoff is you need to decide that you aren't going to do unauthenticated SSR, and I will admit this pretty complicated to set up in its entirety.

reply
I am using Envoy :)
reply
If you're using httpd just fyi - there was a recent RCE vuln for apache that was also http2 specific.
reply
deleted
reply
It's patched so anyone can upgrade to the latest nginx to fix it, thankfully. I agree that just shutting off HTTP/2 is probably non-ideal for a lot of web services.
reply
Neither Ubuntu nor Debian have released patches yet, so if you're using one of these mainstream distributions and install nginx through the distro's package manager, you can't easily "upgrade to the latest nginx" yet. You can however disable HTTP/2 for now.
reply
Both the Debian and the Red Hat packaging systems are awfully convenient ways to build software from source. It has to be, that's package maintainers do daily. They will install all dependencies, apply the distribution specific patches, build, and package in one operation.

Should your distributor fall of the face of the Earth and you still need to patch software, it is often the fastest way to take the source package, put the security patch in the patch directory, and run.

There is more it, such as versioning it between the last release but below the official upgrade so it's convenient when the offical one comes along, and preparing keys if you require signatures, but that's basically it.

reply
At least for Debian, http2 is disabled by default, and so not affected.

You would have had to enable http2 and have a buggy Linux kernel to be vulnerable.

reply
Another reason not to couple your software versioning to Ubuntu and Debians' package managers' schedule
reply
Their package manager and security response is the only reason to use (put up with?) their OS.

If I wanted to build my own software from source every time I'd be running Solaris like a virile man from 2004.

reply
NixOS is probably the best modern option for those who crave building the whole world from source.
reply
yep, already deployed fixes to our fleet (NixOS for the win)
reply
Or just use it in a podman quadlet?

stable-alpine-slim docker iamge is only 14MB?

reply
...or you could use Gentoo.
reply
Keep the flame alive!
reply
Embrace true chaos and let an LLM do the config file merges for you in Portage.
reply
LLMs love gentoo they go hog wild.

Which may be good or bad.

reply
FWIW, for nodes that don't have packages, I compiled from source under /opt and deploy scripts check equal or higher version so it will auto switch back when the new packages are installed. Seemed less risky than disabling http 2.
reply
It makes more sense to install services like nginx via the official containers. In the modern world, the OS is for base services only.
reply
Agreed. At a previous job all the important stuff came from repos that always shipped latest.
reply
The distributions can customize packages with sensible defaults. Debian usually does, and is incredibly convenient. They usually disable features most people don't use, and allow you to easily switch them on if you choose.
reply
You are aware of the massive vulnerabilities coming out across software eco-systems recently right?

Ubuntu and other distros have not kept up.

Podman container auto-updates to the latest upstream. It is a minimal surface area given alpine images. Security first containment.

In this day and age with AI attackers and zero-days it is absolutely daft to still treat servers as the old-school pets model.

reply
> Podman container auto-updates to the latest upstream

Doesn't this make the system more susceptible to supply chain attacks?

reply
This looks like it was patched in Ubuntu on the 12th?

https://ubuntu.com/security/notices/USN-9090-1

reply
This Ubuntu security notice is about an Apache vulnerability, not nginx.
reply
> Several security issues were fixed in Apache HTTP Server.

Um, what does this have to do with NGINX?

reply
It was patched but the docker containers were a few hours behind, I nearly had to build from source on a few servers, until I noticed they were pushing all the tags as I was patching a few hours after the release was cut
reply
I told me agent to do that - similar situation.
reply
> me agent

Did you talk to it like a pirate?

reply
There are large areas of the English speaking world where this is common speech.
reply
New caveman version just dropped! /Pirate
reply
Definitely, but unfortunately I use Debian for my servers so.
reply
> this could've been abused by anyone smart enough to monitor repos like this for exploitable bugfixes for a week

And here lies the risk of AI agents in cybersecurity: the speed of processing for monitoring, scanning, identifying security fixes in open source, writing exploits for affected versions, and attempting infiltration, is a matter of minutes now.

It is not "by anyone smart enough". It is by "anyone ill intentioned enough". They don't need to be super smart anymore.

reply
How's that alignment going...
reply
As much as I like open weight models.. what alignment?

There’s no good solution, it’s a minefield and the only choices are what you’re willing to lose to maybe make it out of it

reply
This is the reason why the oss project i work for is not publishing commits anymore outside of releases but people are freaking out over this.

Cant win.

Seriously reconsidering if open source has a future nowadays

reply
The future is in closed source ecosystems on a modified Harvard architecture
reply
deleted
reply
WTF but I get the point
reply
What happens when somebody tries to contribute based on the last-published commit? Do you merge/rebase their contributions for them?
reply
Haven’t figured out how to deal with contributions going forward…

The issue is that we get dozens of PRs, generating a bunch of code is trivial but reviewing and merging it and making sure it doesn’t break in interesting ways is still pretty hard.

Personally I lean towards not accepting code contributions anymore, but accepting design documents for features and l bug reports etc

reply
How about a strict anti LLM use policy for contributors? It adds what's basically a "proof of human work" challenge to opening new PRs, and effort required to review human PRs usually roughly scales in proportion to the amount of effort required to make human PRs. And human contributors who take time to make a PR manually are probably more likely to become longer term community members and not just drive-by contributors

It's not a perfect solution of course, it just takes OSS back to the fairly broken state it was pre-AI. Still, it might be an improvement and should be at least considered.

reply
We do have a LLM disclosure policy.

I’m not sure how we could enforce an 0 LLM policy in practice though…

reply
Amounts to ‘won’t be getting contributions‘ which is maybe not that bad at this point on the timeline
reply
There's plenty of people who contribute to non-AI open source projects, you just don't get the lazy people who don't want to be a serious part of the project or the people who don't know how to program anymore
reply
I mean, win or not, it has nothing to do with some people freaking out, and that certainly have nothing to do with whether open source have a future.
reply
Not really sure what you’re trying to say; my main point is that OSS is going to change drastically in the next couple years
reply
> Seriously reconsidering if open source has a future nowadays

*In C or C++.

reply
Until we don't replace UNIX clones, Khronos APIs, LLVM or GCC with something else.

C and C++ will stay around for a very long time, even with AI powered systems.

If anything, maybe that is a call that even if implementing compiler backends is hard and time consuming, bootstraping compilers, and new approaches to OS design, are the only path away from such dependencies.

reply
Rust is on its way. Vulkan is no GL, and Gallium it's something else over classical MESA under SGI/Sun.

Legacy 32 bit users with NetBSD and Hyperbola can survive a little with Go based tools/small browsers/mail/usenet/IRC clients and some web browsers with Bubblewrap, but not forever.

reply
Rust being here matters,

https://www.khronos.org

https://vulkan.org/events/vulkanised-2027#program

The only thing closer to it is WSGL.

Also there are much more operating systems than those examples.

reply
rust isn't invulnerable.
reply
A tank isn't invulnerable either, but of someone starts shooting at me I'd rather be in a tank than on a 50 year old bicycle.
reply
Fun fact, Russian soldiers ride on top of their tanks and APCs because its safer than being inside of one during an assault. Doesn't mean you are wrong, but there will always be trade-offs. There is no such thing as perfectly secure.
reply
Tank desant is practiced because there isn’t room in the tank to fit extra people. It’s very much not safer outside of a tank than inside! It’s actually the worst of both worlds, because you don’t have the armor protection but are highly visible. The only advantage it gives over infantry assault on foot is mobility. Which is why it’s generally only practiced when there aren’t sufficient APCs/IFVs available
reply
Fun fact, these specific types of problems aren't possible on a JVM. Yet somehow Java isn't the right language to write a webserver in. These problems are possible in Rust and C/C++. At least its "hard" to have this type of problem in Rust.

Security is hard and is mostly about the number of engineers and amount of time spent looking at a specific implementation. When writing code, do you think about the problems of leaking information due to different operations taking specific amounts of time? That's an actual concern when writing a prod service that uses the network. And it has to be balanced against scalability and issues caused by DOSes. It isn't trivial and it isn't binary...there are always trade-offs. Nothing will ever be perfectly secure and usable and scalable.

reply
These commits weren't visible until today. Yesterday, the last public commit was Nov 9.
reply
How is that done with git?
reply
Just don't push upstream?
reply
How bad are these CVEs? Could any of these be related to some of the recent outages (not claiming, asking) in the last week, coinciding with when this bug was fixed in the repo? Monitoring such repos seems a given for nation states.
reply
RCE is about at the top of the stack for severity.
reply
While that is true, you should also account for exploitability. These exploits are demonstrated in lab environments which are simple and controlled. Random legit traffic will interfere with the exploit logic, so these are very hard to get in practice. Not impossible, but it's also not immediately broken.
reply
According to https://www.rapid7.com/blog/post/ve-cve-2026-123007-critical... it requires 128GB to be transferred in a single connection. "a network slower than approximately 300 Mbit/s will take over the one hour connection limit threshold and will not be able to succeed"
reply
Not all of the bugs disclosed today in nginx require that much traffic to trigger, the reverse proxy HTTP/2 bug found by Xint needs just KBs of traffic.
reply
Thankfully most of these bugs coming out seem very difficult to exploit (likely why they took so long to find). They usually become dangerous when you can build an entire chain of bugs to exploit.

But doing that requires collecting the whole chain. In the current environment we might be finding and patching these smaller bugs before they can be put in to a chain. As long as you are keeping on top of updates..

reply
Great context. Thank you for the add.
reply
put up some bitcoin behind an nginx server and tell hackernews to steal it, see how long it takes
reply
:) If the battering ram is 128gb big, it'll be a long time unless someone has reversed those changes in Nginx to get at the exploit. After that, yeah, it's no contest
reply
This was also true with php2shell but AI got there in the end.
reply
And nginx is so widely used many people will be running it and not really know that. It’s ubiquitous reliable invisible infra for many issues. Like preventing the internet from getting to your RCE riddled application servers, terminating SSL, free software LB, etc. if you have cloud native from the beginning you might not know where AWS uses it but otherwise you have it. Cloud flare I think uses a Rust rewrite.
reply
These days even the "disable X" is enough to get everybody looking in the right direction though. i.e. such hints are (given enough tokens) already RCEs
reply
In my experience, asking an LLM to "Do a security audit" doesn't get you much more than an overview of the most obvious config issues while "look for bugs specifically related to HTTP/2" gives much better results.
reply
Yes, fixing open source software is at most an obfuscated announcement.
reply
HTTP/2 is usually the source of most of these bugs in webservers of all kinds
reply
Oh everybody worth worrying about is monitoring this on every major repo already.
reply
The patch gap.
reply
[flagged]
reply