On the development system, the program would only crash, under a heavy load, on the order of hours (like over 12 hours, sometimes over 24 hours). On the production system, on the order of minutes (usually less than a hour). But never immediately. The program itself was a single process, no threads what-so-ever. Core dumps were useless as they were inconsistent (the crash was never in the same place twice).
I do think that valgrind (had I known about it at the time) would have found it ... maybe. It might have caught the memory corruption, but not the actual root cause of the memory corruption. The root cause was a signal handler (so my "non-threaded code" was technically, "threaded code") calling non-async-safe functions, such as malloc() (not directly, but in code called by the signal handler). Tough lesson I haven't forgotten.
It is not the only tool in the bag. But literally the first question anyone should ask when dealing with any bug is “would attaching a debugger be helpful?”. Literally everyone who doesn’t use a debugger is less effective at their jobs than if they frequently used a debugger.
I use logs and printf. But printf is a tool of last resort, not first. Debugging consideration #1 is “attach debugger”.
I think the root issue is that most people on HN are Linux bash jockeys and Linux doesn’t have a good debugger. GDB/LLDB CLI are poop. Hopefully RadDebugger is good someday. RadDbg and Superluminal would go a long long way to improving the poor Linux dev environment.