upvote
I know the leg I'm standing on is correlation but did we have such widespread outages pre-LLMs? The two that come to mind is that one time some intern pushed a config change that borked us-east-1 and Gitlab deleting their DB. And both had clear comms on the situation within acceptable time frames.

Also, I don't think it's that "AI code is inferior" but rather devs got sloppy and no longer review code. Not to say that everyone's lazy but me---a lot of systemic/environmental factors could be the reason.

reply
Large outages are not a new thing.

People were making memes about the GitHub unicorn a decade ago.

reply
No one is saying they are, they’re asking whether the frequency and scale has changed.
reply
With GitHub specifically, yes. But I remember AWS/GCP/Azure outages were like a yearly occurrence before the 2020s, and we kept having the same conversations back then about infrastructure resilience, multi-cloud, etc. that were too hard to address then and still too expensive to address now.
reply
Maybe one major platform had a major outage annually but AWS certainly didn't have a major outage every year. More like every 3-4 years. I remember exactly two in the last 10 years that impacted the production application I run.
reply
Here’s a timeline of major AWS outages: https://awsmaniac.com/aws-outages/

Just because an outage doesn’t affect your specific app doesn’t mean it didn’t happen. By that logic Microsoft didn’t have an outage last week since my windows gaming pc was fine.

reply
We're not really talking about the same things here. I'm talking about the outages where dozens or hundreds of known platforms are down and it makes the front page of HN and major media coverage. That is the magnitude of many of these recent incidents.

I know there are more incidents than that.

reply
There have been many more than two in the past ten years. Maybe not for you, but depending on the region/AZ, especially us-east-1 as it’s aged, outages are impossible to avoid.

Much better than Azure, but that’s not a high bar.

reply
How many made the front page of hacker news and featured on major broadcast networks? Because that is the level of incident we're talking about here.
reply
It absolutely has. The "why" is a matter of first-party versus third-party (bad AI code making bad services, or good services unable to handle the load from everyone else pushing AI code, or a combination of the two)
reply
> The "why" is a matter of first-party versus third-party (bad AI code making bad services, or good services unable to handle the load from everyone else pushing AI code, or a combination of the two)

Decades of cycles of layoffs and outsourcing, where the core business or organization no longer operates business-critical or organization-critical functions.

reply
Or, the thing that has happened on a huge scale in much less time than decades.
reply
It's the combination of moving faster with less domain knowledge, but AI confidence.
reply
But how do we know it is because of AI vibe coding though?

It could be because some critical infrastructure are due to upgrade after a long delay and this is inevitable at some point.

It could be that AI agentic hacking has pushed for some rearchitecture in faster deployment time, which may result in big outage for industry that has so much legacy and used to big bulk deployment only.

Or it could just be because we rely more on cloud infrastructure than previously.

reply
Failures aren't just a product of AI-written code in the infrastructure, but the infrastructure trying to bear the load of everyone else's AI-written code at massive scale.
reply
The crowdstrike outage was much much bigger, both in reach and in impact. It took days to weeks to recover from.
reply
Remember when AWS messed up their BGP Routes?

edit: it was Facebook, not AWS

reply
But that’s the point around correlation =/= causation. What the parent comment is saying is that the cause could be that attackers have much more sophisticated tools to bring down services. The timing with AI adaption doesn’t prove on its own that lower code quality is the reason.
reply
It's not that AI code is inherently inferior, it's that the bulk of the work engineering does now is code review and the scale of reviews has become exponential which means a lot of stuff just gets punted to agents and not carefully examined by anybody with actual subject matter expertise. Plus automation and platform work (like managing test suites) has also similarly become deprioritized and QA in general is being obsoleted as a field (for worse, really).

This is a systemic issue that's driven by AI but the responsibility lies with leadership who are still operating on the move fast and break things mentality. With everybody swinging for the fences and the median blast radius increasing and the checks/roadbumps eliminated you end up with a lot of incidents because the total number of merges and the size of the diff has increased by quite a bit per sprint.

reply
Also most engineers have always been terrible at reviewing code. They could find most bugs when working through it line by line manually as they wrote the code and the tests, but most could never find bugs in their peer's code through inspection. Now that this - along with business and product understanding - are the primary skills they need, a lot of developers are under-skilled for their job requirements as they suddenly exist today.

Staff-level engineers can do this work because they always have, but if they are suddenly the only ones who can in an org then they are simply overwhelmed.

reply
No the issue with AI code is not that it's inferior but that it solves the wrong problem or doesnt believe an edge case is necessary or other contextual issues unrelated to whether the code is actually correct as written.
reply
I don't know if it's as much as people just hoping that it fails, or the fact that in my opinion a programmer is far more likely to verify code that their reputation might depend on instead of just shoveling in whatever an agent spits out.
reply
Or maybe that cyber attacks are generally known after the fact that they are cyber attacks via things like extortion.

Like take the Binance hack for example. A naughty hacker, instead of stealing bitcoin, could have just caused mass outage and disruption without stealing anything, but didn't because bad actors usually want some financial benefit.

Like if we learn that PII was stolen, then it's be safe to say that this was a cyber attack.

reply
That's just a strawman argument.

While there may be people arguing for that, the more relevant point is that regardless of any possible positives of AI code, there are clear negatives that are being largely ignored. Specifically in such outage cases, the volume of code being put into production with little or no review or understanding.

reply
blaming this problem to vibe coding != ai bad at code

shit programmers have always been a thing, these same shit programmers now putting output from claude straight to production

on top of this there are literally PMs getting involved and doing their own PRs, see vscode recently

reply
> Because people want to believe that AI code is inferior

No, people want to believe that it is good, despite all evidence. However the status quo will not be returning, because it will get better. The catch is that it will somehow end up in your pacemaker or the brakes on your car while it still sucks.

reply
AI code is inferior, but business, the media, and regular people don't care about that. They just want fast and cheap, so you get the slop code you pay for.
reply
I find it fascinating that anyone thinks computer programs will be different than anything else.

Clothes. Food. Appliances. Tools. Electronics.

All of these things got worse, but got WAY cheaper. All of these industries had to shed excess labor like crazy in pursuit of optimization at the cost of quality.

Why would coding be different?

reply
Because you write a program once and then you can sell copies without extra costs per unit you sell.
reply
if only...
reply
>Because people want to believe that AI code is inferior,

This is armchair psychology and extremely bad one because the people who have been concerned with engineering quality have been vastly outnumbered by people who just want to be able to turn their brain off and "ship fast" for probably 15+ years, and that's been reflected in reliability long before LLMs came around.

The code these systems churn out at scale is absolute trash but the financial incentives are to produce out more of it because nobody is ever actually held liable for software breaking in our age

reply
yes, I read some java today where I wished for the clankerd to have done it...

Think system(curl) and random-if-cascades which are never reached - they removed the system(python) in this iteration though...

At this point I think AI-code is just average and probably just fine for corporate environments/SaaS (especially given what thkse produce today), but I still think infrastructure should be a) understood by humans b) produced without installing a backdoor on the dev machine and c) a human should decide what to build and be able to reign in the machine.

reply
it is though. at minimum qa/QC of ai code is way worse. Things are hitting prod right now in many sectors that should never have hit prod. Amazon on mobile (at least android + ff) is unusable ATM as the most recent example.
reply
Just like humans, it varies wildly. Stating anything other than that is speculation at best.
reply
This gets to the implied argument: that humans are inherently better to machine-generated output. The truth is there are a ton of programmers getting paid well to generate crap. Anyone's who been a part of any team that isn't some highly curated A-squad knows this is true. Hell, even teams of 1 know it to be true, when you were rushed, didn't understand the domain or stack, under personal pressure etc.
reply
Except human code is reviewable at a reasonable speed
reply
Before laying off half of my company the last place I worked fired QA first
reply