None of those are really unsolvable problems. I think though the issue it seems everyone in this thread is having is you can't wrap a non idempotent function to make it idempotent no matter how hard you try you have to design your system around it.

> The database is temporarily down. The server catches the exception and records the key status as FAILED.

This is the bug regardless of idempotency, right? It should be recording something like RESOURCE_UNAVAILABLE.