upvote
That’s no excuse. Particularly if it affects people self hosting their workers.

These are folks that routinely make it a point to press on system design and scalability during interviews.

Now they suddenly can’t scale or design systems but we should accept that?

reply
Even if their actions running servers were at 50% load (or even 20%) at the end of 2025, they'd be screwed right now, at this point it's as much a tech/scaling problem as it is a CapEx problem and a construction one. Given the public's responses to AI specific datacenters you can't pick everything.

Even my own systems at home are burgeoning under the load of the my more ambitious hobby projects I'm doing for fun. I don't envy "we had to build five more datacenters to keep up with demand" class problems just from a how many people have to sign off on them perspective let alone the technical difficulty of doing so

reply
They don't hire well at all. You look at who is on their research team and it's people with social science PHDs, not computer science. Looking at the state of the site it's not a surprise. Highly incompetent people have taken over for a while. Let's not forget a member of the original executive team was a sex pest too.

For like 10 years the only feature development they did was by stealing ideas from GitLab. I wouldn't be shocked if little if none engineering discipline has took place at all during this time if it's this brittle to frequent change.

Guessing it was mostly held together with duct tape and poorly written tests/monitoring systems if any other corporate driven software.

We're really going to find out over the next few years which businesses have good practices or not.

reply
Are you kidding? That's an insane amount of load increase to manage. "Scalability" isn't one thing, especially not at that level, so it's ridiculous to knock them for it. That amount of load ripples across your entire infrastructure.
reply
They have the backing of a trillion dollar corporation and can literally hire some of the best talent on the planet. People nor budgets are an issue here.

Why do we keep giving excuses to poor engineering disciplines + poor management? This problem is entirely GitHub's making and acting like it's some unplanned natural disaster is low key pathetic.

reply
As a comparison, Buildkite is now running 1.5B job mins per week without this downtime.
reply
Not really a reasonable excuse considering this is completely broken for self hosted runners / paying customers (for a whole day).
reply
But you’re not self-hosting the tasking service
reply
They couldn't handle sending webhooks for eight hours. That seems more than reasonable to expect.
reply
Because github does not offer the option to (and no, running github enterprise doesn't count).
reply
They don't even let you really run those yourself now either, We moved off a big GHE footprint because the support for it was getting abysmal and features were slowly become GH.com. Ive been dreading the move to GitHub.com and it was worse than I expected.
reply