upvote
Molly Guard

(bookofjoe2.blogspot.com)

In DevOps (and Lean, TPS) the more advanced form of this is the Poka-Yoke (https://en.wikipedia.org/wiki/Poka-yoke). Poka-yokes don't just add safety, they also guide the human away from making a mistake.

The canonical example is the automatic shift knob in a car. The shift knob is designed to 1) prevent you from accidentally shifting all the way back into reverse without pressing the shift button, and 2) prevents you from leaving park or neutral without depressing the brake pedal. This way you don't damage the drivetrain or accidentally cause the car to roll forward/backward.

Poka-yoke is a form of defensive design (https://en.wikipedia.org/wiki/Defensive_design). For a beautiful example of defensive design, see the average electric kettle. If water boils over the top it won't short the device, if it boils dry it'll stop operating, the handle and body are plastic to prevent burning yourself, the handle is ergonomic to make carrying 1.5L of sloshing boiling water not cause you to spill it, the cord is detached from the kettle so you don't yank the cord and spill the boiling water, the switches are located on the bottom away from hot steam, and the lids usually lock while in operation, again to prevent damage from spillage or steam. It's the simplest and safest possible way to boil water, and it's $20.

reply
That page is copied verbatim from https://unsung.aresluna.org/molly-guard-in-reverse/ (which is linked at the top). The original page also has much better formatting.
reply
@dang Can a moderator update the link? The original is much better and we shouldn’t promote the copyposter.
reply
…and Google-hosted.
reply
I once was a communications contractor for the major NJ power utility. One of their long time field techs (let’s just refer to him as Mr. T) was giving a tour of a substation that was built from the looks of it in the 50s. I have, you see, this bad habit of leaning on things… well Mr. T, without missing a beat, slid his forearm between my hip and a faded green Bakelite knob, the kind that goes in and out rather than twisting. He informed me that if I had leaned any further I would have shut off half of Newark.
reply
There's a great piece of software called "molly-guard", which intercepts calls to "poweroff" and "reboot" and similar. It checks if it's being invoked via an SSH session, and if so, it asks you to type the name of the system you're shutting down. That way, you never accidentally shut down a remote server when you meant to shut down your own system (or a different server).
reply
I once accidentally rebooted the reverse proxy for all our production traffic. We got some very quiet two minutes while it came back up.

After that we installed molly-guard with a check for the number of active connections. Made it painless to reboot standby proxies and difficult to reboot active ones.

(We also instituted pairing on production proxy maintenance. I'm not a fan of pair programming but pair maintenance is great.)

I like telling junior hires about this incident because it teaches them that (a) anyone can make mistakes, (b) even serious mistakes usually aren't that dangerous, (c) you can learn a lot from mistakes with the right mindset, (d) we cannot prevent mistakes but with the right system design we can reduce their consequences.

reply
> (We also instituted pairing on production proxy maintenance. I'm not a fan of pair programming but pair maintenance is great.)

It's a great opportunity to share knowledge and techniques and I very much recommend doing so. It's an important way to get people familiar and comfortable with what the documentation says. Or, it's less scary to failover a database or an archiving clutser while the DBA or an archive admin is in a call with you.

Also reminds me of an entirely funny culture shock for a new team member, who was on a team with a much worse work culture and mutual respect beforehand. Just 2-3 months after she joined, we had a major outage and various components and clusters needed to be checked and put back on track. For these things, we do exactly this pilot/copilot structure if changes to the system must go right.

Except, during this huge outage, two people were sick, two guys had a sick kid, one guy was on a boat on the northern sea, one guy was in Finland and it was down to 3 of the regulars and the junior. Wonderful. So we shoved her the documentation for one of the procedures and made her the copilot of her mentor and then we got to work, just calmly talking through the situation.

Until she said "Wait". And some combined 40 - 50 years of experience stopped on a dime. There was a bit of confusion of how much that word weighed in the team, but she did correctly flag an inaccuracy in procedure we had to adress, which saved a few minutes of rework.

reply
I was using my company dev machine via Windows RDP remotely during Covid and installed Glasswire which by default blocks all traffic so I lost access. No one was there to uninstall it so I continued development in my personal machine.
reply
Another fun one is disabling the network interface on a remote server. An acquaintance did that by mistake on a cloud VM running some core services, and the cloud provider had no virtual console for some reason. Ended up having to write off the VM and restore from backup. Fun day at the office.
reply
Long ago, I succeeded once to cut my own access through SSH to a remote server, after some firewall changes. That of course has required a long trip to the server, for physical access.

However that was good, because after that I have always been extra careful at any changes that could affect the firewall in any way. (That is not restricted to changes in firewall rules, because there are systems where the versions of the firewall program and of the kernel must be correlated, so an inconsistent update may make the firewall revert to its default state of denying all connections.)

reply
I can warmly recommend the nohup-sleep-disable-cancel pattern for this, as a dead man's switch for danngerous changes.

https://entropicthoughts.com/locking-yourself-out-with-firew...

reply
I'm reminded of this legendary HN comment: https://news.ycombinator.com/item?id=16530398
reply
I am confused by the second guy who was curious and punched the plastic lid… it says you have to hold the button down for 30 seconds, how did that happen?
reply
The guard itself ends up pushing the button
reply
> There is no worse feeling for a programmer than waking up, walking up to the machine that was supposed to work through the night, and seeing it did absolutely nothing, stupidly waiting for hours for a response to a question that didn't even matter.

No, there's one worse feeling. Walking up to the machine that was supposed to work throughout the night, and seeing it had a surprise update that rebooted the system.

One of my favorite things about ditching Windows.

reply
Fun: the “Molly” in question is Ed Krol’s daughter - he’s the guy who wrote the Whole Internet User’s Guide and Catalog.

https://en.wikipedia.org/wiki/Ed_Krol

reply
I personally know a guy who shut down an oil factory by pressing the molly guard button, just because the button looked interesting.
reply
Oh! Then perhaps the long press required for the iPhone’s action button to trigger is a Molly guard!

Also, perhaps `rm` should be molly guarded to move things to the trash on all systems by default, and delete only if forced to by a flag.

Note: I’d have expected Molly to be a cat, because they tend to be pretty good at disrupting things in my experience.

reply
> Also, perhaps `rm` should be molly guarded to move things to the trash on all systems by default, and delete only if forced to by a flag.

Not all systems, but some (RHEL, I think?) default alias rm='rm -i', yes

reply
disk space is cheap these days alias to mv to trash for an extra layer of protection.
reply
deleted
reply
Seeing long presses implemented for those intermittent and irreversible actions in games is something I‘ve always appreciated. I often end up making errant inputs, especially on keyboards.

A guard I often make for myself is removing/disabling the delete key on my keyboard, and setting FN+Backspace to Delete with whatever control software is involved. I often then repurpose the delete key location to F2, which is typically used to “Edit” a spreadsheet cell or file name.

reply
rm has mollyguarding, that's why every invocation of rm you see on the internet is followed by -f
reply
I think that may be a combination of (IMHO unfortunate) factors:

* Yes, on some systems rm is aliased to rm -i by default.

* Some scripts will use rm -f because normal rm returns an error if the target already doesn't exist but -f doesn't care.

* Finally, sometimes files are just ... I think it's being marked read-only that does it? I've hit this while trying to rm a git checkout; you actually do need to add -f sometimes to succeed. So if you just add -f then it'll always work.

reply
deleted
reply
best molly-guard depicited in "The Good Place": https://www.youtube.com/watch?v=etJ6RmMPGko
reply
Sometimes a pop-up appears that I blindly accept because I happen to be typing something with spaces. Wish that button was protected somehow.
reply
Such pop-ups should never automatically get focused. The increase of them was why I switched away from Windows many years ago, and why I like to root my Android devices. It baffles me that focus-stealing notifications cannot be turned off in most OEM Androids.
reply
I do wish those were a thing on flat touch sensitive induction cooktops! (For all those pesky water droplets causing the cooktop to error out and turning itself off)
reply
I get annoyed even at the thought of those things! Had to use a few while travelling. Ugh!
reply
“Mollyguarding” sounds like a great derogation of unnecessary safety measures. Stop mollyguarding me!
reply
Fun random fact, Eventbrite was first a security company called Molly Guard. I spent years cleaning out the 'mg-' prefixes from the code.
reply
I've been looking for this!
reply
Does the disk drive or sim card slot ejector really qualify as Molly Guard?
reply
The guard is it being a tiny hole you have to find a tool to reach into, instead of a button.
reply
Just please don't start adding molly-guards to your software. The concept only makes sense in the physical world, e.g. where the "important button", that you might never have to press, needs to be in reach all the time. In software, there are better solutions.
reply
my favorite Debian package is Mollyguard so when you shut down a server remotely via SSH it just checks the second time to make sure you really wanted to shut down that server and not your laptop.
reply
"Are you sure?" type guards are not suitable for actions which the user does regularly. If a user repeats this action regularly, they quickly automate the thought process (i.e. don't give it any thought anymore) and it becomes useless.
reply
I agree. Fortunately, molly-guard the software can be configured with automated checks to allow safe actions (e.g. shutting down servers that don't receive significant traffic) without pestering the user.

This means a properly configured mollly-guard is invisible for routine actions but kicks in only when a genuine mistake is suspected because the operation would cause some sort of meaningful loss. That way, users aren't trained to ignore it.

reply
> can be configured with automated checks to allow safe actions (e.g. shutting down servers that don't receive significant traffic)

That's clever. This is what I meant when I wrote, that software allows for better solutions.

reply
Reminds me of this Matt Levine

>> At 08:56 a ‘Trade Limit Warning’ pop-up alert appeared within PTE. This presented the trader with 711 warning messages, consisting of hard block and soft block messages, listed in a single alert where only the first 18 lines of alerts were immediately visible unless the person who received the alert scrolled down. The trader did not appreciate their inputting error and overrode all of the soft warnings in the pop-up.

> You get 711 alerts, you only see 18 of them, you are like “ehh 18 alerts is pretty much the normal number,” you override them all without reading.

reply
Which is why that's not what it does. It asks you to input the hostname instead, just like deleting a repo in Github does.
reply
I know how it works. Please don't nit-pick. It's an interruption that forces the user to confirm. That's what I meant.

I discussed this also here:

https://news.ycombinator.com/item?id=46845740

reply
It's not nitpicking. The nature of the interruption being different is material. I've lost files to automatically answering yes to rm -i y/n confirm. Typing the hostname itself is different enough to get me, at least to stop and go wait, hold on. And snap me out of doing the wrong one. Especially an SSH gateway machine.
reply
[dead]
reply
this isn't like a Molly guard. this is like asking the toddler to be careful
reply