Are these outages caused by introduced bugs, though, or by load issues?

As someone who has spent many years working in high load environments, this is not an uncommon pattern.

You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out.

And then load keeps increasing and you suddenly hit a tipping point where everything keeps failing, and you cant keep up. The things that you thought were perfectly horizontally scalable turn out to have a bottleneck you didn’t even think about until you got to a truly massive scale. Your systems suddenly don’t have the excess capacity to handle load spikes or catchup work, so suddenly any failure cascades and recovery is more and more difficult. You can’t solve the problem with additional hardware, and your perfect scalable design actually can’t scale any more.

This doesn’t have to be about GitHub using LLMs in their code to still be related to LLMs. GitHub gets a lot more commits now because of LLMs and probably get a lot more reads because of LLMs as well.

It could be that the extra usage just pushed them past one of those capacity thresholds.

To me this correlates more to them being bought by Microsoft, a company known for being seemingly incapable of creating quality software to the point that it's not even funny anymore, and also known for sloppifying all the products they touch.

It's caused by human laziness and corner cutting. LLMs can write buggy code or incomplete architectures as much as humans can, but standards have lowered. It's not the LLMs are not capable of also fixing these same issues, but that's additional work.

The funny thing is that if companies wanted they could probably use AI to instead increase uptime. Keep existing QA teams (instead of replacing them) and then use AI for better and more timely monitoring (and messaging even) and as an additional Always-Testing™ layer of QA

They mentioned earlier this year that they were beginning the migration to Azure and that it would take a couple years. I would assume it has more to do with that migration then anything else.

Microsoft acquisition which forced to migrate all to Azure.

I wonder if GitHub actions was a bad idea? Like maybe it's being abused for other kinds of compute besides just builds? And even builds themselves can require a lot of compute. I've only recently had a repo there where I wanted to do builds to make a release (both linux binaries and WASM) and whenever I do that tag and wait a few minutes for those builds to finish I think about all the other projects/repos out there on GitHub doing the same.

I'm really kind of surprised they let us do that - like, why didn't they just have you upload the binaries after building on your local machine?

> We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime.

With what to show for it? If GH did 10x in volume/git commits, it's all LLM sloppy-pasta. Where's the 10x productivity? Where are the amazing apps?

It is definitely and absolutely caused by LLMs. I must do 20x more GitHub operations now, and since the agents know GitHub far better than me, I'm using more advanced features. Multiply this times all of us.

I guess github is kind of a shared garden. Interesting that, like in game theory, if everyone is using it too much, no one gets to use it.

someone from github posted a usage graph from the last year on twitter a while ago and they were serving like 14x more requests in a matter of months. it's frankly impressive they've kept up.

It's probably not attributable to AI in the way that you're thinking - Github has been absorbing an exponential increase in usage, and that increase is mostly due to new AI-related projects being created and worked on.

Though I'm sure some of the blame can go to internal slop code.

The last month GitHub hit four 9s of uptime was November 2024

For the longest time, I didn't appreciate the "AI-induced traffic" excuse. But seriously, I checked the rough github egress for our lab versus an old log from 2024, and there's an order of magnitude or two difference. From asking around, it seems people all have the gh cli tool and let it loose with parallel tool calls and e.g LLM's polling Actions in a background bash while loop with sleep $TOO_FEW_SECONDS. Some people use a variety of skills where the agent makes a commit every few code changes, and uses Issues for its memory/log. And they have O(5) sessions at the same time doing all kinds of crap. It's the same with PR checks/PRs. Recently we also saw continued usage throughout the night as well, which did not exist pre coding agents. Loops or whatever they call cron jobs in the harnesses these days is the reason. It must be adding up.