upvote
I'm not 100% informed on this, but it seems very unbelievable to me that Envoy could be considered more robust or widely used than nginx at any point in history including today. I started using ConnectRPC instead of GRPC almost entirely because it didn't leave me subject to (what I consider to be) an unnecessary dependency in Envoy.
reply
I guess I would just say, wordpress is also more widely used than many other projects, but that doesn't necessarily make it more robust or secure than the projects Google, Cloudflare, and security companies develop.

The main reason I see to not use Envoy is that it's very annoying to set up and build, and requires familiarity with things like grpc/proto/low level networking to get right. But if you're using grpc and http2 I think you're generally past that "infra overhead" threshold already and it's going to better serve your needs.

I have pretty much only worked at big tech or my own infrastructure startup, and did some systems/infra work at Google using Envoy for a while. I can't say it's 100% better than nginx but I can say that I've evaluated nginx and a lot of software like it (ie with a similar customer base, ecosystem, and history; like coturn) over the years for my personal use, and every time I've immediately been annoyed by constant security issues or poor http2/L4 networking support that I don't see when I use what Netflix/Google/etc use.

reply
Envoy probably has the advantage that it’s only a reverse proxy server, unlike nginx which has other uses too like a general-purpose web server with optional modules.
reply
You can implement custom http filters for envoy or use this for general web serving: https://www.envoyproxy.io/docs/envoy/v1.15.2/configuration/h...

The documentation, DX/UX, and discoverability for these features are pretty bad. But they work well if you know how to do it.

Implementing a custom HTTP filter itself is actually easy, I can share examples I did in the last year for a voice inference stack if anybody is interested. The problem is just that the devloop is very painful, because Envoy doesn't make it easy to know why something isn't working (this is actually most of what I did with it at Google, tons of testing/verification and compatability infra for relatively simple http stuff) and the builds take a very long time.

reply
On all my small projects (eg rails apps) the reverse proxy to the app servers is the same server expected to serve the static assets directly. In fact the majority of bot nonsense never goes through the proxy due to path rules.

Envoy is only attractive at a higher scale.

reply
Does Envoy support things like rate limiting?
reply
Yes [0]. For a single node you can just use this, it works like the Golang/language rate limiters you might already be familiar with [1]

In a multiple node/distributed architecture you need away to coordinate rate limits across nodes and not try to fight the CAP theorem.

There is a way to do this [2] [3] and although it sounds complicated the shape of the solution (delegating to a third party service that aggregates and provides client-level rate limit metrics across multiple proxies) would still be the same any way you cut it.

[0] https://www.envoyproxy.io/docs/envoy/v1.15.2/configuration/l...

[1] https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/extensio...

[2] https://www.envoyproxy.io/docs/envoy/v1.15.2/intro/arch_over...

[3] https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/config/r...

reply
I don't know what nginx security vetting looks like, but just being older isn't a security guarantee. I know Google spends a lot of manpower on making envoy secure.
reply
What WAF would you suggest to use with envoy? Have been working with Coraza and tbh satisfying is something else. So i wonder what someone that recommends envoy would recommend as WAF for it
reply
Personally I serve all static, public content off a CDN and only serve dynamic content through Envoy after it passes AuthN/AuthZ.

I use JWT-based sessions + a remote JWKS (which is remote static content that Envoy can cache) + either a third-party IDP or dedicated login service (this being the only application logic besides Envoy's routing exposed to the public Internet). That way, every envoy auth check is immediate and stateless.

In this setup you don't really need a WAF because all unauthenticated requests redirect to signin or 403, which get strict rate limits on login. An abusive peer that sends a ton of traffic anyway gets blocked at the network/firewall level outside of Envoy. The tradeoff is you need to decide that you aren't going to do unauthenticated SSR, and I will admit this pretty complicated to set up in its entirety.

reply
I am using Envoy :)
reply