I use some fairly popular (in the MSP space) backup software that thinks the network is infallible. The worst case I’ve seen is when it fails on a network request, doesn’t retry adequately, and incorrectly logs the error as data corruption.

Let's say you want to retry a network request. It's... A bit more complex than it seems, right?

Firstly, you need to know exactly what type of error you ran into. Some errors aren't really recoverable. Maybe a programming issue occurred and you are constructing an invalid URL and the HTTP client is yelling at you. No sense in retrying that 20 times. Maybe it's a network error, that seems like a good candidate to retry. Maybe, the request succeeded and we have a response, but it is a 500 error, again, seems like a good candidate.

Secondly, you need to know if it is safe to retry. If the request is essentially idempotent, like a read-only GET request, then surely it is safe, right? But, what if it isn't safe? Forget about solutions like idempotency tokens; let's assume you don't control that. Now you need to figure out how you can know if the request had side effects. If a well-known 4xx error is returned you might know, but if you get a network error or a 5xx error it's much harder. Did the request fail during a buffered response after the side effects were already applied? Maybe you can check to see if the request applied with another request. Now you have two network requests, and both need error handling.

Finally, and probably most obviously, you have to make sure you don't hammer the server when it is under load. To avoid the thundering herd problem, you'll probably want to use an exponential backoff with some jitter.

What sucks about all of this is that while there are reusable components here, the concerns effortlessly cut through different layers, making them a pain in the ass to deal with. It isn't that it is impossible for a library to handle all of these problems (I anticipate an excited evangelist may reply explaining how their favorite library does it all in one package if this post gets enough visibility) it's just that this is hard and these problems repeat in different forms, in a way that makes it difficult to fully eliminate the repetition. And this is just the most obvious basics, whereas in reality there are almost always case-specific complexities.

You can, for example, encapsulate a reasonable exponential backoff with deadline implementation and apply that as appropriate for different things, but you can't really cheat your way out of having to think about all of these things, especially if you don't control all of the network APIs you might have to interface with.

This is one part of why I don't like try/catch exceptions. They are an appropriate mechanism to use as a failure isolation boundary due to their stack unwinding capability: it would still be bad in most cases if a logic error or upstream error not being handled properly in a single network request handler were able to crash an entire network server, so being able to blanket catch everything that bubbles up an log it is good. But then using this for normal error handling, it makes doing the wrong thing perhaps just a bit too easy. I don't think you should have to self-flaggelate in order to say "just crash if this errors", but I do think that you should have to say it. Try/catch exceptions are backwards by default, just write normal looking control flow and no errors are handled and it's hard to tell if there even are any. Checked exceptions try to fix this but somehow this feels even worse; now you have a flattened list of exceptions that may occur at various different layers of depth, in some cases the same exception can occur at different layers of depth, you may literally need to read source code and map out the call stack in your head to be sure. (Hope it doesn't change later.)

The Result or Expected type concept seems like the way to go in the frame of modern programming languages. Go's error passing also works OK though it has papercuts (that a linter can help you with, at least.) To me it makes more sense to make stack unwinding error handling a more niche feature used for isolating error domains, rather than use them for all error handling.

But even that! Even that doesn't solve the problem. You still have to sit there and think about the types of errors that can occur and their consequences. At best, explicit error handling with value types just encourages you to confront it and makes it visible, even in cases where you still say "OK, pass to caller".

If you flip it and instead ask "how do I write something that can't fail?" you might find some interesting ground.

The best things I know about are static type-checking, pure functions and totality. Different languages provide more or less help with these things. It's perfectly fine to do 'two things which don't fail or cause other things to fail'.

Forgive the digression, but there is an 'infectious' aspect to the above 3 things (see the function-colouring problem), e.g. you can't build pure functions which call non-pure functions. The Dependency Inversion Principle (of SOLID) gives some help in how to tackle this.

Also, the above things only work within one node (of a distributed system).

For multiple nodes, I use something like Kafka, where you write down one event, and have two systems subscribe to it, each doing one thing. Yes, there's still the obvious issue of them failing independently, but when that happens, you have an authoritative source of truth (in the form of Kafka events). This beats the craps out of developer logs.

You skip the laborious questions of "what happened in the system?" and "what should the correct state be?" Because the events are already the answer - just eyeball them.

Events also machine-readable, so if you diagnose a problem and a fix it in one case, there's a good chance you can build a detector for other cases. You don't have to wait for a support ticket to get escalated to the dev team.

You also divide the debugging space dramatically. If the Kafka log says one thing {Bob bought Minecraft for $10}, then the Ownership service is just wrong if it says Bob doesn't own Minecraft, and the Finance service is just wrong if it doesn't report the $10. Fix each independently. At no point do you need to look at Ownership and Finance together to see which one failed halfway through talking to the other, because they don't talk to each other.

Lastly, events are verifiable; they are their own audit trail. If your boss asks how much money is in the system, would you feel more confident reporting whatever the current balance is set to (i.e. the outcome of whatever code executed the last "UPDATE Balance ..." statement, or would you like to be able to sum over every transaction that you ever recorded?

Now they have at least two problems.