upvote
There's literally no such thing as "black start" at Microsoft that I am aware of. The way services are brought online initially is through a build out process coordinated through hundreds of Teams. It's an organic, messy, and painful way of doing things. As such, build outs are a human driven process with bits of automation spanning sometimes 20+ years of engineering touches, troubleshooting guide, and some additional tribal knowledge thrown in passed down through the years. There is little consistency across teams and sometimes even within a team as every Azure Cloud is effectively slightly "different".
reply
I never worked in civilian power generation, but I can vouch for this on Nuclear Ships.

We had to practice load pickup on a normal start at the pier often, but we likely practiced a full SCRAM to back online (including a FULL load shutdown) more.

Having predefined lists of what gets brought up at various points helps a ton, but it's still not an easy or simple operation.

reply
I imagine the predefined list is easier when it's all consumers that are under your control. I figure with civilian power some random percent of houses'll kick their AC on after power is restored.
reply
Back in 2025, Spain did a very-dark-brown-almost-black start (with help from neighbours), starting with one hydropower plant and increasing from there.

It went remarkably well.

reply
> Black starting a service is obviously quite hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster.

I mean... you can try, but unless you spend a lot of resources on it, I don't expect test load to be close enough to really practice a black start.

From experience, the best way to prepare for restoration from nasty outages is to have them frequently! The team I was on was pretty good at outage restoration, until we started regularly hitting about a year between outages. After a few years of one or maybe two outages per year, we had lost the muscle memory for protecting services that get overloaded during or after an outage and setting outage messages (which both informs customers and protects customer service queues).

Regularly starting up an isolated cluster at least lets you get ahead of sequencing issues though; hopefully it's enough if you can get the whole cluster started and then modulate inbound traffic (most complex services will need something like you mentioned so that you limit the number of sessions, but sessions that start are likely to proceed successfully).

reply
It's more testing theater on their end; it's just perfunctory rubber stamping. There was and is no real performance testing under simulated near-real world conditions.
reply
Did you do anything to try and fix that?
reply
Yes. As another commenter said, you can partition the clients into pods-clusters. And use your test suite software against a realistic pod-cluster. Azure has such a test suite: https://learn.microsoft.com/en-us/azure/app-testing/load-tes...
reply
Somewhat related: https://company.sbb.ch/en/railway-development/services-rus/e...

SBB just finished their project to be able to restart the grid in 5min. (There was a famous multi hour outage a few decade ago that I think motivated this work)

reply
3-5 days in the UK according to "regulations". Some talk of it potentially taking 7-14 days under certain bad circumstances.
reply
The biggest challenge of bringing a resilient system back online after an outage, is the resiliency of said system fighting back.
reply
Texas was on the verge of having to do a black start a few years ago when winter weather started causing failures. I think people were quoting potentially weeks to get everything back online. As it was, I think people didn't have power for days during sub-freezing temps due to load shedding.
reply
deleted
reply
Good to know even Google introduces circular dependencies!
reply
Happened to Spain last year. The black start.
reply
In software servers this is called the “thundering herd” problem. It is why clients are coded do not just exponential backoff but random offsets too.
reply
It's also a well known problem, so no excuse for that.
reply
a well known problem with more than a few known fixes, one of which is load shedding your upstream based on protocol aware load balancrs.

I've had similar issues where backends couldn't come up due to load killing them as they came up: my solutions was to turn off the load balancers until the backends all came up before opening the door again...

I agree: no excuse.

reply
deleted
reply
Not only did they not handle thundering herd, it happened across all regions.
reply
deleted
reply
that's not the problem with grid black start at all? black start is difficult because many types of generators themselves need power to start. it's a circular dependency problem, not a load management one.
reply
It’s both, but every grid has a sub-set of power stations with black-start capability (a crap ton of diesel generators), that exists to break the circular dependency problem and bootstrap the rest of the grid.

But load management while bootstrapping the rest of the grid is still very hard. Often the reason for really long blackout, ones measured in days rather than hours, is repeated cascading failures as operators try to reconnect parts of a the grid, and accidentally overload recovered parts of the grid, causing generators to trip off again.

reply
That was a very enlightening moment in Factorio when I had already converted everything to power and the whole grid blacked out and I needed to get it going somehow again.
reply
Well stated.
reply