upvote
In another life I worked as an engineer commissioning oil rigs and I’ve seen how tricky even a small-scale black start can be. On a rig, we simulate total power loss and have to hand-crank a tiny air compressor just to start a small emergency generator, which then powers the compressors needed to fire up the big ~7MW main generators. It's a delicate chain reaction — and that's just for one isolated platform.

A full grid black start is orders of magnitude more complex. You’re not just reviving one machine — you’re trying to bring back entire islands of infrastructure, synchronize them perfectly, and pray nothing trips out along the way. Watching a rig wake up is impressive. Restarting a whole country’s grid is heroic.

reply
I can appreciate the ability to revert to hand cranking an air compressor, yet I can't help but feel that the 99.99% of events, you'd be better served with keeping a two stroke gas engine ready to go. Air compressors tend to have parts just as or more vulnerable to environmental factors, and you get a lot more power for less elbow grease out of a two stroke.
reply
The fewer resources we dedicate to grid resilience and modernization, the harder black starts become. And as grids get more complex and interdependent, recovering from total failure becomes exponentially harder.

A rare but sobering opportunity to reflect on something we usually take for granted: electricity.

We live in societies where everything depends on the grid — from logistics and healthcare to communications and financial systems. And yet, public awareness of the infrastructure behind it is shockingly low. We tend to notice the power grid only when it breaks.

We’ve neglected it for decades. In many regions, burying power lines is dismissed as “too expensive.” But compare that cost to the consequences of grid collapse in extreme weather, cyberattacks, or even solar storms — the stakes are existential. High-impact, low-frequency events are easy to ignore until they’re not.

reply
Just to highlight this: the last significant power outage in Western/middle Europe was 2003. [1]

That's 20 years without any significant problems in the grid, apart from small localized outages.

It's not hard to start taking things for granted if it works perfectly for 20 years.

Most people don't even have cash anymore, either in their wallet or at home. In case of a longer power outage a significant part of the population might not even be able to buy food for days.

[1] https://en.wikipedia.org/wiki/2003_Italy_blackout

reply
Honest question, are we better off in the long run, and is it a better solution, to decentralize energy generation and make more smaller grids rather than linking them all up? This isn't to say completely getting rid of the ability to transfer between the smaller grids to assist with power disruptions but to decouple and make it less likely for catastrophic "global" failures like this.
reply
It's hard to build big generators, so as we already struggle with infrastructure I don't think that's feasible, but it would be great.
reply
Yeah, this is the turkey’s dilemma - life on a farm is a lot better than life in the wild for 51 out of the 52 weeks of the year.

Most of our modern economy and systems are built to reduce redundancy and buffers - ever since the era of “just in time” manufacturing, we’ve done our best to strip out any “fat” from our systems to reduce costs. Consequently, any time we face anything but the most idealized conditions, the whole system collapses.

The problem is that, culturally, we’re extremely short-termist- normally I’d take this occasion to dunk on MBAs, and they deserve it, but broadly as a people we’re bad at recognizing just how far down the road you need to kick a can so you’re not the one who has to deal with it next time and we’ve gotten pretty lazy about actually doing the work required to build something durable.

reply
Interestingly it seems that the black start drill is considering a smaller zone of impact than what has happened here.

Also I suspect there is far more renewables on the grid now than in 2016.

This is potentially the first real black start of a grid with high renewable (solar/wind) penetration that I am aware of. Black starts with grids like this I imagine are much more technically challenging because you have generation coming on the grid (or not coming on) that you don't expect and you have to hope all the equipment is working correctly on "(semi)-distributed" generation assets which probably don't have the same level of technical oversight that a major gas/coal/nuclear/hydro plant does.

I put in another comment about the 2019 outage which was happened because a trip on a 400kV line caused a giant offshore wind farm to trip because its voltage regulator detected a problem it shouldn't have tripped the entire wind output over.

Eg: if you are doing a black start and then suddenly a bunch of smallish ~10MW solar farms start producing and feeding back in "automatically", you could then cause another trip because there isn't enough load for that. Same with rooftop solar.

reply
The frequency aspect of a black start is presumably a bit easier in Europe because there's an interconnected synchronous grid so they can bootstrap it from France essentially.

It's far more problematic for the UK because all the interconnects are DC.

reply
I was recently told by an electrical engineering lecturer that the black start plan here in Ireland is to use the DC interconnectors with the UK to provide startup power to a synchronous generator.
reply
With the new wexford-wales interconnect that went live last month, and another one planned from Cork (?) to France things might be even easier in the near future I reckon.
reply
Did he mention the UK's plan in case they have a full blackout?
reply
Does the source of "truth" matter? You still have to turn everything off, bring your generators up, each in sync and then allow load back on slowly.

The UK keeping its own time just makes things easier for it IMO.

reply
A synchronous interconnect provides not just a source of truth but also stabilizes your grid frequency. If you have an isolated grid you have to match generation to demand to keep the grid frequency stable. If you have a 1GW interconnect that means you can mismatch generation and demand by up to a gigawatt and still be fine. I imagine that makes for a much faster startup procedure
reply
If the isolated grids all each synced to a source of truth first (GPS?), would it be possible for them to immediately connect afterwards?
reply
Practical Engineering did a really great video a few years ago on why black starts are hard, complete with a tabletop demo about the physics of synchronizing large spinning generators: https://www.youtube.com/watch?v=uOSnQM1Zu4w
reply
It seems Spain lost 15GW of load, but is still running 10GW of load: https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh...

Would this suggest the grid hasn't snapped apart, or is it just not possible to tell from the data?

Coal, pumped hydro, and nuclear generation all went to 0 around the same time, but presumably that's those sources being disconnected from the grid to balance demand? https://transparency.entsoe.eu/generation/r2/actualGeneratio...

reply
They're definitely doing a black start:

https://x.com/RedElectricaREE/status/1916818043235164267

We are beginning to recover power in the north and south of the peninsula, which is key to gradually addressing the electricity supply. This process involves the gradual energization of the transmission grid as the generating units are connected.

I see load dropping to zero on that graph, or rather, load data disappears an hour ago.

If the grid frequency goes too far out of range then power stations trip automatically, it's not an explicit decision anyone takes and it doesn't balance load, quite the opposite. A station tripping makes the problem worse as the frequency drops even further as the load gets shared between the remaining stations, which is why grids experience cascading failure. The disconnection into islands is a defense mechanism designed to stop equipment being too badly damaged and to isolate the outage.

reply
BBC reporting the head of Spain's electricity grid saying restoring power could take "between six and ten hours": https://www.bbc.co.uk/news/live/c9wpq8xrvd9t?post=asset%3A85...
reply
Interesting, but in terms of load I think think the data may just be delayed by ~1 hour. Switching to UTC, to avoid timezone confusion, it's currently 13:10:

Last actual load value for Spain at 12:15: https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh...

Last actual load value for France at 12:00: https://transparency.entsoe.eu/load-domain/r2/totalLoadR2/sh...

reply
It may also be a measurement artifact. Looking at the generation by type page,

https://transparency.entsoe.eu/generation/r2/actualGeneratio...

Everything dropped to zero except wind and solar, which took huge hits but not to zero. I expect those have been disconnected too, as they cannot transmit to the grid without enough thermal plant capacity being online, but if the measurement at some plants of how much they're generating doesn't take into account whether or not they were disconnected upstream they may still be reporting themselves as generating. You can't easily turn off a solar plant after all, just unplug it.

Either that, or they're measuring generation and load that's not on the grid at all.

reply
Probably they are estimates of not grid metered generation assets based on wind speed and solar production, at least in the UK nearly all solar is 'estimated' because it is not measured directly (apart from larger sites), at least in real time.

Rooftop solar for example just shows as a reduction in demand, not 'generation' per se.

reply
> Thermal plants require energy to start up

It's not just about the power. System components cannot be brought to operating temperatures, speeds and pressures faster than mechanical tolerances allow. If a thermal plant is cold & dark, it can take days to ramp it to full production.

reply
That's true of some kinds of thermal generators, but not all. Simple cycle gas turbines can come up very quickly (think jet engines). Or your car's engine.
reply
Would this be relevant here? Spain is connected to the French grid, and probably also Morocco.

The entire EU runs on one synchronised grid so from that perspective a single 'province' went offline, not the grid.

reply
It would be essentially the same thing as a grid black start, except that the first breaker to close has the European grid on its primary side, instead of a freshly started generator under your control.

The complex process of configuring the transmission network to bring grid power to each power plant in succession is the same.

reply
Does solar power make this process easier or harder? I know that with thermal plants you have a spinning mass that you have to synchronize, and phase shift is used to assess how hard the plant is working (and whether to trip a disconnect as we see here)

But with solar, how is the synchronization provided? In like a giant buck? Or in software somehow? Does the phase shift matter as much as in the electromechanical systems?

My intuition is that solar would make the grid harder to keep stable (smaller mass spinning in sync) but also may offer more knobs to control things (big DC source that you can toggle on/off instantly.. as long as sun is out). But I don’t actually know.

reply
Most solar and wind plants follow the inertial lead of the thermal plants. They can't synchronize without enough thermal generation being online. Supposedly there are efforts to change that, I don't know enough about grid engineering to say how far along that might be in Spain.
reply
Mike_hearn's comment was grey but was correct: phase following is indeed done through software in the inverter. Phase matching is still required, wherever the phase difference is not zero there is a deadweight loss of power as heat.

Currently the main driver of battery deployments is not so much energy price time arbitrage as "fast frequency fresponse": you can get paid for providing battery stabilization to the grid.

(for the UK not Spain: https://www.axle.energy/blog/frequency )

reply
Oh that is fascinating.

So if you have a smarter solar panel, or a smart battery, you can stabilize the grid. I’m assuming that all of the traditional software complexity things in distributed systems apply here: you want something a little bit smart, to gain efficiency benefits, but not too smart, to gain robustness benefits.

My intuition is that bringing the market into it at small timescales probably greatly increases the efficiency significantly but at the cost of robustness (California learned this “the hard way” with Enron)

> Phase matching is still required, wherever the phase difference is not zero there is a deadweight loss of power as heat

If the electronic controller is “ahead of” (leading) the grid, then that heat would come from the solar plant; if it is “behind” (following) then that heat comes from the grid. Is that right? And likely, solar plants opted for the simplest thing, which is to always follow, that way they never need to worry about managing the heat or stability or any of it.

I wonder if the simplest thing would be for large solar plants to just have a gigantic flywheel on site that could be brought up via diesel generators at night…

reply
I would think that renewable infrastructure could be the fix, at least if you start installing larger battery capacity to meet renewable store and usage shifts, the grid essentially is installing the resources that can also be used to respond to & contain sudden source losses and prevent cascades.
reply
Ukraine can share some experience with forced black starts.
reply
cash is all well and good but if the tills dont work most stores wont be able to serve you. corner shops, vape shops and barbers will be ok though
reply
[flagged]
reply