upvote
If you're using grpc, http2, etc you're so much better off using Envoy proxy

You basically get to piggyback off of a large chunk of big tech's infrastructure and security work. Obviously everything has the risk of security flaws (and I kind of hate that default Envoy builds ship with half-baked filter impls) but it's a generally more mature ecosystem than projects with a lot of smaller/legacy users like nginx

reply
I'm not 100% informed on this, but it seems very unbelievable to me that Envoy could be considered more robust or widely used than nginx at any point in history including today. I started using ConnectRPC instead of GRPC almost entirely because it didn't leave me subject to (what I consider to be) an unnecessary dependency in Envoy.
reply
I guess I would just say, wordpress is also more widely used than many other projects, but that doesn't necessarily make it more robust or secure than the projects Google, Cloudflare, and security companies develop.

The main reason I see to not use Envoy is that it's very annoying to set up and build, and requires familiarity with things like grpc/proto/low level networking to get right. But if you're using grpc and http2 I think you're generally past that "infra overhead" threshold already and it's going to better serve your needs.

I have pretty much only worked at big tech or my own infrastructure startup, and did some systems/infra work at Google using Envoy for a while. I can't say it's 100% better than nginx but I can say that I've evaluated nginx and a lot of software like it (ie with a similar customer base, ecosystem, and history; like coturn) over the years for my personal use, and every time I've immediately been annoyed by constant security issues or poor http2/L4 networking support that I don't see when I use what Netflix/Google/etc use.

reply
Envoy probably has the advantage that it’s only a reverse proxy server, unlike nginx which has other uses too like a general-purpose web server with optional modules.
reply
You can implement custom http filters for envoy or use this for general web serving: https://www.envoyproxy.io/docs/envoy/v1.15.2/configuration/h...

The documentation, DX/UX, and discoverability for these features are pretty bad. But they work well if you know how to do it.

Implementing a custom HTTP filter itself is actually easy, I can share examples I did in the last year for a voice inference stack if anybody is interested. The problem is just that the devloop is very painful, because Envoy doesn't make it easy to know why something isn't working (this is actually most of what I did with it at Google, tons of testing/verification and compatability infra for relatively simple http stuff) and the builds take a very long time.

reply
On all my small projects (eg rails apps) the reverse proxy to the app servers is the same server expected to serve the static assets directly. In fact the majority of bot nonsense never goes through the proxy due to path rules.

Envoy is only attractive at a higher scale.

reply
Does Envoy support things like rate limiting?
reply
Yes [0]. For a single node you can just use this, it works like the Golang/language rate limiters you might already be familiar with [1]

In a multiple node/distributed architecture you need away to coordinate rate limits across nodes and not try to fight the CAP theorem.

There is a way to do this [2] [3] and although it sounds complicated the shape of the solution (delegating to a third party service that aggregates and provides client-level rate limit metrics across multiple proxies) would still be the same any way you cut it.

[0] https://www.envoyproxy.io/docs/envoy/v1.15.2/configuration/l...

[1] https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/extensio...

[2] https://www.envoyproxy.io/docs/envoy/v1.15.2/intro/arch_over...

[3] https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/config/r...

reply
I don't know what nginx security vetting looks like, but just being older isn't a security guarantee. I know Google spends a lot of manpower on making envoy secure.
reply
What WAF would you suggest to use with envoy? Have been working with Coraza and tbh satisfying is something else. So i wonder what someone that recommends envoy would recommend as WAF for it
reply
Personally I serve all static, public content off a CDN and only serve dynamic content through Envoy after it passes AuthN/AuthZ.

I use JWT-based sessions + a remote JWKS (which is remote static content that Envoy can cache) + either a third-party IDP or dedicated login service (this being the only application logic besides Envoy's routing exposed to the public Internet). That way, every envoy auth check is immediate and stateless.

In this setup you don't really need a WAF because all unauthenticated requests redirect to signin or 403, which get strict rate limits on login. An abusive peer that sends a ton of traffic anyway gets blocked at the network/firewall level outside of Envoy. The tradeoff is you need to decide that you aren't going to do unauthenticated SSR, and I will admit this pretty complicated to set up in its entirety.

reply
I am using Envoy :)
reply
If you're using httpd just fyi - there was a recent RCE vuln for apache that was also http2 specific.
reply
deleted
reply
It's patched so anyone can upgrade to the latest nginx to fix it, thankfully. I agree that just shutting off HTTP/2 is probably non-ideal for a lot of web services.
reply
Neither Ubuntu nor Debian have released patches yet, so if you're using one of these mainstream distributions and install nginx through the distro's package manager, you can't easily "upgrade to the latest nginx" yet. You can however disable HTTP/2 for now.
reply
Both the Debian and the Red Hat packaging systems are awfully convenient ways to build software from source. It has to be, that's package maintainers do daily. They will install all dependencies, apply the distribution specific patches, build, and package in one operation.

Should your distributor fall of the face of the Earth and you still need to patch software, it is often the fastest way to take the source package, put the security patch in the patch directory, and run.

There is more it, such as versioning it between the last release but below the official upgrade so it's convenient when the offical one comes along, and preparing keys if you require signatures, but that's basically it.

reply
At least for Debian, http2 is disabled by default, and so not affected.

You would have had to enable http2 and have a buggy Linux kernel to be vulnerable.

reply
Another reason not to couple your software versioning to Ubuntu and Debians' package managers' schedule
reply
Their package manager and security response is the only reason to use (put up with?) their OS.

If I wanted to build my own software from source every time I'd be running Solaris like a virile man from 2004.

reply
NixOS is probably the best modern option for those who crave building the whole world from source.
reply
yep, already deployed fixes to our fleet (NixOS for the win)
reply
Or just use it in a podman quadlet?

stable-alpine-slim docker iamge is only 14MB?

reply
...or you could use Gentoo.
reply
Keep the flame alive!
reply
Embrace true chaos and let an LLM do the config file merges for you in Portage.
reply
LLMs love gentoo they go hog wild.

Which may be good or bad.

reply
FWIW, for nodes that don't have packages, I compiled from source under /opt and deploy scripts check equal or higher version so it will auto switch back when the new packages are installed. Seemed less risky than disabling http 2.
reply
It makes more sense to install services like nginx via the official containers. In the modern world, the OS is for base services only.
reply
Agreed. At a previous job all the important stuff came from repos that always shipped latest.
reply
The distributions can customize packages with sensible defaults. Debian usually does, and is incredibly convenient. They usually disable features most people don't use, and allow you to easily switch them on if you choose.
reply
You are aware of the massive vulnerabilities coming out across software eco-systems recently right?

Ubuntu and other distros have not kept up.

Podman container auto-updates to the latest upstream. It is a minimal surface area given alpine images. Security first containment.

In this day and age with AI attackers and zero-days it is absolutely daft to still treat servers as the old-school pets model.

reply
> Podman container auto-updates to the latest upstream

Doesn't this make the system more susceptible to supply chain attacks?

reply
This looks like it was patched in Ubuntu on the 12th?

https://ubuntu.com/security/notices/USN-9090-1

reply
This Ubuntu security notice is about an Apache vulnerability, not nginx.
reply
> Several security issues were fixed in Apache HTTP Server.

Um, what does this have to do with NGINX?

reply
It was patched but the docker containers were a few hours behind, I nearly had to build from source on a few servers, until I noticed they were pushing all the tags as I was patching a few hours after the release was cut
reply
I told me agent to do that - similar situation.
reply
> me agent

Did you talk to it like a pirate?

reply
There are large areas of the English speaking world where this is common speech.
reply
New caveman version just dropped! /Pirate
reply
Definitely, but unfortunately I use Debian for my servers so.
reply