I once worked at a company that had a wealth of backups. A backup generator, backup batteries as the generator takes a few seconds to start, a contract for emergency fuel deliveries, a complete failover data centre full of hot standby hardware, 24/7 ops presence, UPSes on the ops PCs just in case, weekly checks that the generators start, quarterly checks by turning off the breakers to the data centre, and so on.
replyIt wasn't until a real incident that we learned: (a) the system wasn't resilient to the utility power going on-off-on-off-on-off as each 'off' drained the batteries while the generator started, and each 'on' made the generator shut down again; (b) the ops PCs were on UPSes but their monitors weren't (C13 vs C5 power connector) and (c) the generator couldn't be refuelled while running.
Even if you've got backup systems and you test them - you can never be 100% sure.
A plan that has never been executed is really just hope and wishful thinking.
replyWhat happens when the backup breaks?
replyAt a certain point earth is a single point of failure.
replyYou have a back up for the back up backup.
replyTurtles all the way down.
At AWS scale even unlikely hardware events become more common I guess.
Each turtle gives them another 9. How many 9s are they down due to incidents over the past year?
replyThey're definitely more than half a day now, which is only two and a half nines.
replyThey absolutely have backups, I presume they were ineffective or also down for _reasons_.
replyThe point of being "cloud native" is you build redundancy at higher levels. Instead of having extra pipes and wires, you have extra software that handles physical failures.
reply