upvote
Snark aside, this is actual advice in Crisis Engineering by Nitze, Weaver, and Dickerson, and largely matches my experiences as well. You can harp on and on and on at upper management about why budget or attention needs to be allocated to take care of something before it blows up into a much bigger problem, but they'll never move on it -- until it actually does blow up, and then suddenly you've got nearly a blank check to do whatever's necessary to make sure it never happens again.

If you don't take full advantage of that, you're missing out on possibly your only opportunity ever to fix whatever's needed fixing for years.

reply
That makes sense, and yet...

If things blow up because you're upgrading to something supposedly better and it goes wrong, then that's not a time when you can properly fix the new thing. You can only push forward with the transition or go back to the old. I disagree with "you've got nearly a blank check to do whatever's necessary to make sure it never happens again." You have a blank check, but anything you use it for has to be scoped within stopping the bleeding. As soon as you're onto the new or back to the old, the check expires and you likely can't touch anything real anymore because it'll be involved in lawsuits and finger-pointing.

I don't disagree with the core message, but I think it plays out differently for a major upgrade vs a production meltdown on an existing system. With mostly an existing system, you can pour effort into resilience. With an upgrade, I doubt people would let you beef up a system that is still on the chopping block, even if it's getting a life extension. And if you're still switching to the new system, all effort is going to go into getting it up and running and anything that gets in the way of that (as quality improvements are likely to) is going to face some serious resistance. The check is not truly blank.

reply
> first reaction

The FAA et al. have been working to modernize US ATC since ~2003. https://en.wikipedia.org/wiki/Next_Generation_Air_Transporta...

The next step is a ton of hardware upgrading, which always costs real money.

reply
> working to modernize US ATC since ~2003

I was working on upgrading US ATC in 1995. As were lots and lots of other people.

reply
When the system is held together with chewing gum and 1980s-era software, when it fails, "someone screws up" is not a fair reaction. Of course it fails! The wonder is that it still mostly works.
reply
> The wonder is that it still mostly works.

A lot of very hardworking people getting too little pay for something so critical while their employers (some contracting company) take a lot of profit. Eventually they burn out waiting for help and a replacement system and retire or quit, then the systems really start failing because the knowledge leaves with them.

reply
Should probably have waited 48 hours for the news cycle to move on and ensure absolutely nobody knows the money is needed
reply
They asked for the money before the crisis which makes the timing even MORE conspiracy coded somehow
reply