upvote
On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.
reply
I've been in AWS for almost twenty years at this point. It's been a long time since I've seen a global outage of the data plane on anything. The control plane, especially the US-east-1 services? Yes - but if you're off of east-1, your outages are measured in missile strikes, not botched deployments.
reply
Didn't the latest outage affect people not on us-east-1 because internal aws services depend on us-east-1?
reply
The impacts are usually partial. For example, scaling is impacted but everything already deployed contributes to work up to capacity. Or, you can't change configuration but the previous configuration works as configured. Often surprisingly not so impactful even if there can be limited work stoppage.
reply
The problem with the us-east-1 outage is that a lot of big companies are there, so even if you try your best not to depend on us-east-1, your third party providers are most likely there. In my previous company, we were completely down during us-east-1 outage because of other dependencies that are beyond our control.
reply
Entirely fair. I have thus far avoided that problem. Not always engineering's choice.
reply
Work for a major bank who isn't solely in US East 1.

No it didn't impact us.

reply
Perhaps you don't notice GCP outages because so few companies rely on them?
reply
GCP has a lot of customers. But you wouldn't know the companies that do, unless you worked there and wanted to leak it, or it publicly comes out. Eg it's been publicly acknowledged that Apple uses GCP for iCloud, https://www.cnbc.com/amp/2018/02/26/apple-confirms-it-uses-g... , and Home Depot is another that's used as a case study, https://cloud.google.com/customers/the-home-depot but most customers don't want to make a big deal about being on GCP as it's none of our business who's hosting them.
reply
Apple also uses AWS, and I won't be surprised if they also use Azure. Big companies are multicloud, and not because it's a good idea (it rarely is), but because they inherited multiple environments on different CSPs, and maintaining those where they are is often cheaper than migrating them to a different CSP.
reply
I wonder if big companies can get a special contract with something like you can't delete my service automatically (unless it's an emergency)
reply
upvoted & favourited because you taught me a really interesting fact which I feel makes up for an amazing discussion (regarding icloud using GCP).

also, I can't help but imagine if instead of render, it was Apple's account which could've been auto-banned (Render is almost a billion dollar company or series-B, I am not sure)

I haven't read the articles and I admit that but can you please elaborate to me on why Apple uses GCP themselves for idrive, I would love to know the technical decisions behind it on a genuinely curious level.

From my (let's face it) limited understanding of GCP, it isn't particularly good or price performant and one of the wonders is that Google sells it directly with Google photos too and an competitive lineup at android.

So in some sense if Apple is using gcp's for icloud then aren't they just reselling google storage themselves and google can always beat them in pricing while also wanting to chew away at the percentage of iphones themselves too?

I mean, I can still try to understand the google search pays apple 10 billion dollars (right?) deal but I don't quite understand why apple would pick GCP when the hosting market is one of the more competitive ones with lots of companies.

I would love to get some explainations or theories as to why exactly is that the case

(Also given its HN, if anyone from apple is reading or knows the answer, I would love that too!)

reply
> So in some sense if Apple is using gcp's for icloud then aren't they just reselling google storage themselves and google can always beat them in pricing while also wanting to chew away at the percentage of iphones themselves too?

Apple uses Samsung displays and Sony camera sensors, iirc, both of which are flagship Android phone makers. That doesn't really seem to be a concern in their procurement thinking. iCloud and Google Photos are not that direct competitors because which one is native depends on which phone you already bought. Google Photos definitely does have some market share on iOS due to having 3x the free storage and a handy compression mode (which used to be entirely unmetered at launch but now still uses storage, just less of it). But it will never be a full competitor because it is a separate app you have to install and it can't magically fetch cloud-only photos from the camera roll and photo picker UI like iCloud can.

The pricing of Google One and Apple One/iCloud+ isn't really dictated by underlying storage costs. At the higher tiers like 2TB, many don't come close to using all, while the laughable 5GB iCloud free tier clearly costs almost nothing in raw store, even on nVME SSD, if you compare it to S3/Backblaze or even raw disk pricing on the cloud.

reply
GCP has had outages. From a quick search it looks like they had a global outage less than a year ago:

https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

reply
GCP never goes down because they banned all their customers.
reply
AWS goes down catastrophically but are back up in minutes/hours most of the time (as long as they aren't down because Iran blew up their data center). That's obviously REALLY bad for certain industries, but I suspect for the vast majority of their customers it's not a big deal. We've been able to isolate the damage almost every time just by having AZ failover in place and avoiding us-east-1 where we can.
reply
> AWS goes down catastrophically but are back up in minutes/hours most of the time

The outage in the linked article appears to have been resolved in 4-5 hours.

reply
IIRC the Paris datacenter flood took down a whole “region” and some data was permanently unrecoverable.
reply
>On the other hand i can’t remember when there was a serious outage on GCP

They had a really bad global outage a year ago. At least with AWS outages are contained to a single region.

reply
How is blackhole-ing a customer not considered an outage?
reply
Unfortunately, if everyone goes down people are understanding. If just _you_ go down, then its oddly less forgiveable.
reply
There was a pretty bad one last summer - their IAM system got a bad update and it broke almost all GCP services for an hour or so, since every authenticated API call reaches out to IAM.

It had lasting effects for us for a little over 3 hours.

reply
You can read the parent post, right?
reply
I still remember the one where they nuked all the storage of I think an Australian insurance company I think, luckily the it department had done a multi cloud setup for backups
reply

  Google Cloud accidentally deletes $125 billion Australian pension fund - May 2024
https://www.business-standard.com/world-news/google-cloud-ac...
reply
https://en.wikipedia.org/wiki/Timeline_of_Amazon_Web_Service...

Azure nerfed the front door of all Azure and O365 services last year.

All of these companies are great at what they did, and occasionally fuck up.

reply
> Never heard of this from AWS or Azure.

AWS does it more efficiently; it takes down many startups at a time when us-east-1 goes down.

reply
That’s an entirely different type of problem, and avoidable by just using us-east-2 (I still don’t understand why people default to us-east-1 unless they require some highly specific services).
reply
Is it that easily avoidable? A lot of AWS's control plane seems to have dependencies on us-east-1, or at least that's what it's looked like as a non-us-east-1 user during recent outages.
reply
I don't know how much it's improved, but a bunch of URLs they use unnecessarily have region specific details in them.

I remember a Workspaces outage about 5 or 6 years ago, and the problem for us was that the redirect link in the console had US East 1 in it.

The workspaces themselves weren't in US East 1 and nothing relied on US East 1.

Emailing users who needed it an alternative link with a different region in the URL for the login redirect fixed it for us.

reply
Sympathy. Railway is going to have numerous people blaming them for this outage. When us-east-1 fails, it is headline news, so you are not to blame.
reply
During my 5 years of my startup, we had only 1 outage due to AWS because we picked us-west-2 as the primary reason. If anyone starting a company and picks us-east-1 as the primary reason, they should be fired. There's absolutely no reason to be in that region.
reply
Why do people want to be in that region? Is it the default or something?

I know some workloads help to be colocated but all these places are connected by fiber and every cloud has a worldwide CDN it seems.

reply
If my cloud provider brings my startup down, it's my problem. If they bring all the startups down, that's their problem.
reply
And we all celebrate it since we can't do any work
reply
Yep, we also don't touch them for this same reason.
reply
Yep, agree 100%. Such a stupid move on their behalf.
reply
Hetzner and OVH also do this all the time.

It's AWS and Azure that are the outliers and tend not to care too much what their customers do with their infrastructure. AWS is perfectly fine with allowing me to run copies of 15 year old vulnerable AMIs copied from AMIs they've long since deprecated and removed. Even for removed features like NAT AMIs.

reply
AWS normally contacts you first.
reply
Do they?

The only anecdotal thing I've seen is we hired a vendor to do a pentest a few years ago, and they setup some stuff in an AWS account and that account got totally yeeted out of existence by AWS if memory serves.

reply
You should not be conducting unauthorized penetration tests against third party infrastructure providers without permission. They have processes and systems and usually just wants a heads up of what you plan to test and t the duration / timestamps.

Cuz otherwise you look like a threat actor.

That’s assuming your vendor was pentesting AWS systems. If you meant you hired a vendor to pentest your own systems on AWS, that’s of course a totally different matter.

reply
>That’s assuming your vendor was pentesting AWS systems. If you meant you hired a vendor to pentest your own systems on AWS, that’s of course a totally different matter.

Sorry for being unclear, the vendor was attacking our organization only, and any other company was expressly forbidden in the contract. As I recall it was a fake SSO sign-in page to collect credentials that they would try and social engineer our employees with.

reply
At a minimum you should contact AWS before you launch a phishing page as a test that targets AWS customers.
reply
I’m fairly certain you are supposed to contact any vendor before attempting to penetrate hosts with authorization, not the other way around.
reply
Having done this for both Azure and AWS, there's a specific ticket that needs to be filed with each provider that documents the scope of your pen test, where you're coming from, and a time frame over which you're doing it (which ISTR was "not more than 24 hours")
reply
Responding to an unknown security tester like that is a selling point, not a cautionary tale
reply
Yup, I thought it was great. Although one concern I always had in the back of my mind was where is the line drawn. Such as if an adversary gains access to one of my orgs accounts and does something similar, do we get 100% taken out.
reply
deleted
reply
They better do. What is google doing?
reply
It's all AI powered
reply