upvote
What they can implement is to slowdown the commit rate, rate limt or just queue-up messages not to overburden their downstream service.

I don't think GH has any of those, but just keep scaling, but that scaling failed.

Just bad architectural decisions from the postmortem.

--

It will only get worse due to AIs spawning massive commits, and they don't have unlimited cloud resource.

They can scale but not scalable in terms of effort, resources, and $

reply
How would any of what you're saying help with this?

> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery.

reply
This stuff is good! This is what a booming economy looks like. There are people out there competing with you for resources because they have cool ideas they want to implement.
reply
Or at least they asked the AI to come up with cool ideas, which is even more interesting. It's exciting watching the world transition away from humanity being in the driver's seat!
reply