upvote
And this is Railway, a big enough name to top the HN main page and presumably find someone from Google to intervene at some point. I would have zero recourse if it was some little product that I built.
reply
Their account was restored in 10 / 19 minutes! It just took 4-6 hours to get everything fully healthy. I look forward to seeing the google response to this hopefully.

May 19, 22:10 UTC - Our automated monitoring detected API health check failures and paged our on-calls, who started investigating the issue. May 19, 22:11 UTC - Dashboard returning 503 errors. Users unable to log in. May 19, 22:19 UTC - Root cause identified: Google Cloud Platform has suspended Railway's production account. May 19, 22:22 UTC - P0 ticket filed with Google Cloud. Railway's GCP account manager engaged directly. May 19, 22:29 UTC - Incident declared. May 19, 22:29 UTC - GCP account access restored. All compute instances remained stopped and persistent disks inaccessible.

reply
100% agree, I've seen on Twitter and HN small players facing similar issues with no recourse and response from Google. I don't know what kind of place they are trying to build there.

They got TK to woo the enterprise customers who were forced to be hostage to OCI. But it seems they are still doing opposite of hostage here.

reply
This is the bigger point of all of this. Scary.
reply
At least from the outside TK seems to be doing well given GCP's growth. My completely uninformed assessment is that he stepped in as the disciplined adult in the room to override Google's otherwise lackadaisical approach to enterprise. (Clearly still some ways to go, as this incident shows.) Now, that may have created a culture that is at odds with the rest of Google, but it was probably required to become a "serious" enterprise org.

That said, did OCI, being an Oracle division, have a culture worth destroying? On the other hand, I could see TK importing that culture into Google...

reply
GCP was never known for their support and deprecation of services was always a huge risk. Its very sad because its actually a quality product. They should easily be the number 2 provider. Azure is extremely unreliable and their documentation is subpar. GCP being in 3rd place is more of their doing.
reply
i wouldnt really call it a product without support, let alone a good product.

its a nicely design hobby, that somebody could make a good product out of, by following the same abstractions

reply
> Don't expect Google quality from the name.

It sounds exactly like what I have experienced in terms of Google quality over the decades.

reply
All google products work like this. Should never be used for anything critical.
reply
Yeah, sadly know that from being burned from one of their depreciations. In fact, 2-3. But you live and you learn. And it is better to learn from other's mistakes always.
reply
Hasn't theGoog acted this way of quick to suspend accounts well before Gemini? I like to bash on LLMs as much as the next guy, but this seems very much like the memory of a gold fish. Or, you are just too young to remember pre-LLMs???
reply
Haha, no. I know Google bans anyone randomly with usually no recourse in sight. I just wanted to take a dig at how bad their LLM is too while we were at it and thinks like Google themselves which is not surprising.
reply
You missed the humor part and focused on the tech part, it seems.
reply
They probably do use similar tech to make some of these decisions, though. And they always have done that as well.
reply
> Don't expect Google quality

Google has an extremely poor reputation. Why are you thinking differently to that?

reply
What/who is TK? What is OCI?
reply
Thomas Kurian (GCP ceo)
reply
OCI is oracle cloud infrastructure.
reply
That Google poached from Oracle Cloud, fwiw.
reply
This is just typical anti-Google FUD. Been on GCP for nearly a decade and as do my peers. Sure, you hear about a few stories like these and in this particular case with Railway, I would actually wait to see what caused the trigger for the suspension - both from Railway and from GCP. But, this sort of thing happens with every cloud provider including AWS (you can google for the same thing "AWS shut down our account with no warning") and you'll find tons of stories like these.

As a former GCP consultant, I can share that these sort of shut downs aren't random and it's usually due to the customer not being compliant - that breaks cloud compliance requirements for the big clouds, so automated systems flag it. Eg. Someone serving CP on their CDN, for instance.

The Railway incident report also doesn't directly address this at all other than:

    May 19, 22:22 UTC - P0 ticket filed with Google Cloud. Railway's GCP account manager engaged directly.

So, I would actually like to know more (What did the account manager say exactly?) before I just simply jump onto the Google hate train because it's cool to do so.
reply
This feels like google applying the same anti-spam mindset everywhere: detect risk, ban first, ask questions later.
reply
It's pretty stupid that big customers like Railway are not somehow protected from this.
reply
I think all customers should be protected by at least one CSR doing a quick approve before banning the account.
reply
That seems to be the case. But as we see it backfires. Railway is very public, but we know at hacker news google has been doing this kind of thing for quite some time now.
reply
yep, last year they deleted UniSuper private cloud too
reply
> It seems like they use Gemini 3.1 Pro to run their production decisions.

They said they are already using Gemini 3.5 Pro internally.

reply
Then it's a bad endorsement for Gemini 3.5 pro too. But jokes aside, I think they need a customer centric thinking instead of a self-centred one they seem to harbour even before TK joined (not everything can be blamed on him although it should be his responsibility now).
reply
Google? Customer-centric? The closest thing to that is their cloud division buttering up some big name clients.

Other than that, Google prefers to act like "customers" are some kind of unfortunate rash they can't quite seem to get rid of, but would love to do so.

reply
Yup, updated with the article I mentioned by Steve Yegge. Still holds true today.
reply
deleted
reply