That's a common bias in management. Heroic fire fighting gets rewarded while silently preventing a crisis with proper planning isn't. Nobody ever gets credit for fixing problems that never happened.
Someone hit on the notion that it’s Stockholm Syndrome. You bond with people you go through shit with. You don’t bond with people who prevent the shit. That was a super awkward conversation. The asshole in charge quit shortly after it, but 2 years later we still occasionally called each other out for making plans that accounted for a toxic chain of command that was no longer toxic.
Exactly why the general public thinks Y2K was a "hoax" or "overblown".
Unfortunately, that typically meant when you were paged, the problem was probably significant, hard, or it was another team and you were just being triaged in because you were affected or might be able to help root cause.
This anecdote hits differently for me. I grew up reading about Fog Creek and consuming all of their writings (and that one film) as an example of what a software development company should look like. It feels strange to have the curtain peeled back to reveal that something like backups, which would be considered basic requirements today, were not part of their early operations.
My company during Sandy was operated in both NY and SF, and our experience was a bunch of us on the west coast filling in for those who were not only unable to get to the office, but without power themselves in NYC. It was all unfamiliar stuff for us (different business functions were done there, but the NYC office’s work was time critical) but did build camaraderie.
"Everyone coming together to solve a problem by any means necessary" still loses something if the problem is sufficiently imaginary.
Like, carrying fuel up 17 flights to keep a children's hospital powered on, or a MUD that's been running continuously since '93? Sure, inspiring as hell.
"Our company's product will be offline for a few days and we'll theoretically lose money/marketshare, but all our customers will be super understanding because they were all watching the news and, holy shit, did you see that tunnel get flooded, and then everything will turn back on and be fine with no extraordinary effort on our part...?"
In the case of Fog Creek during Sandy, because the software we hosted provided planning and project management for a lot of people, going offline would have a pretty large ripple effect. Sure maybe 95% would've been fine, but I knew of customers who e.g. managed emergency services and crisis response, and so keeping them online was meaningful to me.
This was a startup. The company getting fucked could mean our jobs going away. It was our equity too.
Not everybody is just punching a clock, while resenting the owners of the company. In our case, we wanted our company to do well because we liked working together.
If MUD is the type of game you're falling into the same trap as the one you're signaling but worse. You're putting in the same sentence "saving kids' lives" and "the bragging rights for a game's uninterrupted session" on equal footing.
When we went offline customers were super understanding!
Except one customer who was outraged and we felt totally comfortable banning their account.
If I flip a fair coin and don't look at it as it sits in my hand, the probability of heads is 50%. Your estimate should be the same. Before or after the flip doesn't matter because our state of knowledge is the same.
If I look (and you don't), the probability that it is heads from my state of knowledge is 100% or 0%. Your estimate would be unchanged because you didn't get any more information.
If somebody looks at the coin and tells you what they say, you estimate should be somewhat less than 100% or more than 0% depending on how much you trust them.
All of this is the long way around the barn to say that after the hurricane, the probability that Sandy hit Manhattan is definitely 100%. That's past tense probability, however. It is still also true that the best estimate a year before it hit would be much, much lower. Using words like "probably that a hurricane would hit Manhattan" makes interpretation much harder, of course, because of the uncertain time frame.
Nobody talks like this in real life. I can't even believe someone bothered to write multiple sentences to try and justify it.
It really was a shocking wake up period for me in the realities of the real world vs. how we often perceive things.
IIRC it was the whole "Wasabi" thing that kind of opened up my eyes that there was a lot more smoke and mirrors involved than I had realized before.
So many times it's taken herculean effort to make things happen, usually because of poor planning and poor leadership.
Why these are viewed as "successes" or "heroic" baffles me as well.
It's not a mystery, it's exploitation.
With all the business efficiencies we've made with computers as devs and IT, we should all be working 3 days weeks, instead a few people are so rich they can buy elections.
It's never going to work like that, and you don't need any dastardly, sinister, or conspiratorial explanations for why. It's a natural side effect of the market being competitive.
When tools and technology make people more efficient, some people take that to mean they can work fewer hours and get the same job done. Which is true. But other people take that to mean they can work the same amount of hours, and get way more done. This is also true.
In a market in which two companies are competitors, and one takes the former approach and one takes the latter, which company do you think will still be around in ten or twenty years? Obviously the latter. And that pressures people in the former group to demand more output and stable working hours so they can remain competitive.
That is an argument of faith (that companies where employees work longer survive more than companies that optimise differently), and I am not convinced it is supported by evidence.
One could argue that all other things being equal, companies with employees doing more hours are more productive (even though I seriously doubt that, there are diminishing returns very quickly). But then I would argue that if a company drives their employees harder, there are leadership and management issues, so other things are not equal. And this is associated with younger workers (adaptable: good, inexperienced: less good), high turn-over (so, long-term information retention issues) and lower worker satisfaction (so the whole thing is self-defeating in the long term).
Besides, there are many, many factors involved that have nothing to with with working time. Again, leadership, strategy, operations, marketing, personal connections, dumb luck…
Personally I think there is an optimum somewhere, and at some point you get more out of your workers by making them more invested and motivated than making them work longer.
But everywhere I have seen people overworked has terrible morale and low productivity in spite of the hours.
I think it also remains to be proven that the company that works their employees the hardest will necessarily outcompete the others.
It’s not necessary that the longer hours actually translate into productivity, just that the boss feels like it should.
Well, good organizing, too.
Yet after 100 years of explosive technological growth we still have the labouring hours of the 1930s...
The result was that business owners invested in their workers and long term health of their business so it would continue to give them 10 mm per year instead of trying to cash everything out every 1/4.
Your source of confusion seems to be not recognizing that sometimes in life multiple things can be true at once. "Heroism" is generally about the actions of the individuals in response to a dangerous scenario, not how the scenario came about in the first place. "Success" is more overloaded, but can refer to both the overall system in place and the outcome of a given dangerous scenario, again separate from what caused that scenario to come about. This is near perfectly true in particular if, as is usually the case, the ones dealing with merde-hitting-spinning-blades on the ground are not the leadership. It may be idiotic, short sighted, lazy, inept leadership that lead to the problem (or it may simply be an unforeseen alignment of holes in the cheese), but if people on the ground go above and beyond to prevent the worst that's still heroic of them, personally. And if their efforts work that's a success. It'd be pretty strange to insist that the efforts of firefighters shouldn't be recognized just because somebody else did something stupid that resulted in a fire.
Yes, it's absolutely just as important what comes after. Ideally it's like the commercial aviation industry, where heroism on the parts of flight crews can be recognized and said heroism results in a top-to-bottom full systemic investigation and efforts to ensure that ideally nobody will ever be heroes in quite that way ever again. And have constant efforts to make sure heroism isn't ever required in the first place.
The opposite extreme of some organization using it as an excuse for inaction and cheapness (at least until the day arrives when heroic efforts are not enough and a true disaster really does occur) is abominable. Even worse, because in that sort of utterly failed leadership environment the people on the ground will probably get scapegoated too. And it's 100% worth acknowledging that there can be a "success trap" and Normalization Of Deviance situation here, where heroics keep averting the worst which lets bad leadership/management escape correction. But it's a trap precisely because each individual failure can still be bad, up to and including literally people dead, so the pressure to succeed is still there each time.
But if employees, be they civilian or military for that matter, perform far beyond the call of duty why would that ever not be heroic?
Anyway, I was the first Fog Creek employee on site at 55 broadway (one of our founders got their first, and two peer1 employees were also on site) and also part of the first wave of the "bucket brigade". Another dozen of us showed up over the coming hours as we assessed and fixed things.
The media coverage on this was never super accurate-- we were carrying (and later, relaying) 5 gallon covered plastic buckets (like the home depot buckets with sealed lids) up to the landing where the generator was to top up its reserve tank. At one point SquareSpace (who was also running exclusively out of this DC) hired a group of muscle to bring a 55 gallon drum up, but that wasn't us.
I don't view this as lionizing failure, though I can see your perspective. Across our products (then FogBugz, Kiln, Trello) we had different backup strategies in play, but we were definitely putting a lot of eggs in the local-datacenter-that-claimed-to-have-resilient-power basket. Kiln in particular had decided to use SANs for data resilience, and that proved particularly troublesome for trying to find a workaround once the storm hit-- FogBugz on the other hand was able to spin up an EC2 version and start a slow-but-would-have-worked-eventually migration (and we even had an employee hit the road for us-east-1 location with a bunch of encrypted backups so that we could get a fast data restore online), and Trello fast-tracked-and-completed its local-to-cloud migration. While it was true that the DC had a backup generator, the fuel tanks for the generator being in the basement and becoming unavailable was not part of their resiliency plan. I don't fault us for not catching that.
...although I did have to later write the dreaded "I'm sorry, we've been hacked" blog post as a direct consequence of us not properly deprovisioning a "temporary" AWS credential that we created during that sprint. I will fault us for not catching that.
Shortly after Sandy, I switched my role from core dev on FogBugz over to DevOps, and I built out the backup and restore resiliency we had been missing. It wouldn't have kept us online during Sandy, though.
It was dangerous, and a couple folks slipped and fell. That's not a good thing and I won't celebrate that. Many employees opted out of the bucket brigade for family, safety, or personal reasons. I threw out all the clothes I wore that week as they reeked of diesel... not great.
I wouldn't wish that experience on anyone, but having come through it with fortune and no slip-and-falls, I still think about:
- Walking around FiDi post storm and seeing the flooded tunnels, clogged with cars that had been washed into them from the street parking.
- Dropping off big armfuls of coffee and bagels in the lobby for the building work crews. The folks who were essentially dredging out the basement and getting the building back online did much harder work than us.
- Going shopping for aquarium supplies in order to best jerry-rig the pump that our founder has repurposed from the office fish tank in order to transfer diesel from the drums on the curb into our 5g buckets.
Something that kept me going was that I knew that one of our FogBugz customers was the emergency response service in Haiti, and they got absolutely walloped by the storm. They were using FogBugz to coordinate relief, so there wasn't any way I was going to let FogBugz go offline if I could help it. Turns out that meant climbing a lot of stairs.
The issue that I remember was that we had so many SQL Server databases that it was impossible to keep anything vaguely approaching a sane backup strategy; SQL Server just isn't designed to handle as many as we had. (I in fact remember talking to a famous SQL Server consultant [edit: this was likely Brent Ozar] we were using who said "that won't work," and then I or someone else was tasked with talking to a SQL Server dev at Microsoft about it and they ALSO said that it couldn't work, and when I brought this up, I was told that Microsoft was wrong. IIRC, the implication was that it would take an extremely long time (many days) to restore, and that the snapshots would likely be days old. I don't think any of that contradicts your memory about the very a slow restore going on, but might add some color.
(Likewise, Kiln supported multiple backend instances and had admin commands to copy and move repos between them, so it the SAN had specifically been the issue, we had workarounds that would've been fine with whatever EBS was up to those days. But I remember the databases being the blocker.)
Our "One database per tenant" model was also the reason we were so slow to go to AWS-- the MSSQLServer licensing terms were cost-prohibitive.
However recognizing that around 90% of the FogBugz databases were free-tier and didn't get daily changes meant that teaching the backup scripts to skip creating redundant backups solved most of the issue.
Post-Sandy, I was able to solve most of this by:
1) Shipping full, diff, and incremental backups at different schedules
2) Scripting the ability to create warm standby servers powered by the latest fulls, then diffs as they come online, then the logs at the cutover time. (so you can provision a new server and seamlessly cut over to it)
3) Using our own scripts instead of the RedGate tool (which was _fine_ just not equipped to handle as many DBs as we had) to orchestrate the backups and restores. This in particular was needed for the "only back up modified dbs" fix.
This got us to most of what we wanted-- 0-downtown transitions of databases across servers, nightly offsite backups, fast backup runs. The final problem-- if our datacenter evaporated and we had to reprovision everything from scratch -- was left as an acceptable risk of "we'll fetch the nightlies from s3/Glacier while we do the slower part of rebuilding our servers."
But to your point, during Sandy, none of the above had been fixed yet. I think our sysadmin team was very strong, but they weren't programmers (and/or didn't have enough time to step back and program), and I was able to pick a lot of low hanging fruit in DevOps
There is no reason that they couldn’t have had more than 1 DC when Sandy hit.
Not everyone is built the same, thankfully.
I guess it helps that I grew up in a rural place and did construction for a while, so lugging gas to a generator wasn't a new thing for me.
Also I was... 25? I'd be more cautious today, I think.
Firefighters are heroes who bravely enter burning buildings, beloved by children and adults alike, muscular figures who get pictured on sexy calendars and thanked by tearful families.
Officious paper-pushers who make a fuss about a few cardboard boxes stored under the fire escape stairs (like me!) seldom get pictured on sexy calendars.
A firefighter entering a burning building is literally risking his or her life at that moment. A paper pusher who works on preventing the fire in the first place is normally very far from danger.
Im all for doing some dirty work when needed, but if they expect 100% uptime they should be ready for something as common as a storm flood without risking injuries hauling fuel drums by hand.
If you drop back to 30m of lift, you get still have 36PSI which is enough to get some flow. But that means that you need a holding tank (or lots of buckets) and another pump for another 30m of lift. That gets you to the 16th floor so you need another holding tank and a another pump.
That's a lot of complication. And getting those pumps to sit still can be a challenge.
Or you could label it Hyrox/Crossfit/$FITNESSTREND and charge people money to do it. #barrelpump #verticalmarathon #fitnessgoals #blessed
Quitting a job because of this is so funny to me. 2012 was truly a different time.
My story [not as interesting] is that I was a emergency responder living in BPC and was relocated to the millennium hilton because it had a generator... BPC never lost power :/
For once , the name fits lol
If there was electricity to run the window washing crane, they wouldn't need to haul diesel.
You can do a barrel knot, though with a modern metal barrel, you can also generally just do a hook on top an bottom and lift horizontally.
My comment was more about the goofy shit people do to make the failure mode simpler next time instead of eliminating the failure mode all together. Some of the places I've worked would probably have made one of us get trained on the lift or something instead of make sure we have the fittings on had or a fuel pump and hoses with 17 floors of head pressure. That would be the temporary fix and then when they got the quote for the pump it would become the fix.
The chance of a hurricane within 24 months was never 100%. Looking back, it did happen, but you can’t judge future-facing decisions on post-hoc information that you didn’t have at the time.
There is a difference between “no one wanted to admit it wasn’t 0%” and “no one else thought the risk-adjusted return was positive.” When people decline to take some preventative measure, that’s usually what they’re saying implicitly.
I’ve been where you are—frustrated that you predicted and told people exactly what would happen, they didn’t listen, and you were right—many times in my career. It’s infuriating. Part of learning from it is figuring out how to get people to change, not merely figuring out how to be right.
My guess is that this kind of black/white Strong Opinions approach has bitten you throughout your career.