Not printf exactly, but I've found bugs with a combination of mprotect, userfaultfd and backtrace_symbols when I couldn't use HW breakpoints.
Basically, mark a set of pages as non-writable so that any writes trigger a pagefault, then register yourself as a pagefault handler for those and see who is doing the write, apply the write and move on. You can do this with LD_PRELOAD without even recompiling the debugee.
Note: I do work at MSFT, I have used these capabilities but I’m not on the debugger team.
https://learn.microsoft.com/en-us/windows-hardware/drivers/d...
i haven’t seen this type of capability used in too many companies tbh and it seems like a lot of opportunity to improve stability and debugging speed and even code exploration/learning (did i break something ?)
A concrete use case for this is catching memory corruption. If your program corrupts a known piece of memory, just set a hardware watchpoint on that memory address and BOOM, the debugger breaks execution on exactly the line that's responsible for the corruption. It's a fucking godsend sometimes.
Vscode (or an editor with ADP support) supports unconditional breakpoints, watchpoints, and logpoints (observe & log values to the debug console).
printf debugging works in all of them, even the build system and shell scripts.
A debugger can be great, no question about it, even for remote debugging. In my experience, I’ve seen fewer people effectively debug unfamiliar systems quickly.
In particular, REPL tools will work on remote session, on pre-production servers etc. _if_ the code base is organized in a somewhat modular way, it can be more pleasant than a debugger at times.
Makes me wonder if the state of debugging improved in PHP land. It was mostly unusable for batch process debugging, or when the server memory wasn't infinite, which is kinda the case most of the time for us mere mortals.
I use IPython / JShell REPLs often when the code is not finished and I have to call a random function without entrypoint.
In fact its possible to jump to the graphical debugger from the Python REPL when running locally. PyCharm has this feature natively. In VSCode you can use a simple workaround like this: https://mahesh-hegde.github.io/posts/vscode-ipython-debuggin...
Kind of a hybrid of logging and standard debugging. "everything" is logged and you can go spelunk.
For example:
The ”debug” package on npm is something in between, as it requires inserting debug statements but they are hidden from output unless an envvar like DEBUG=scope.subscope.*,otherscope is used.
pdb is great for python, though.
I simply use conditional break points to break when whatever go routine happens to be working on the struct I care about.
Is there more to the issue?
- Things like "continue", "step" are no longer a faithful reproduction of what the program does in real time, so it's more difficult to understand the program's behavior. Some timing-related bugs simplify disappear under a debugger.
- There's usually some background thread that's logging things to console, which reduces to problem 2 in my comment.
I haven't used Go that much. I imagine since goroutines are such a cornerstone of the language, the go debugger must have some nifty features to support multi-(green)-threaded debugging?
VS, on the other hand, gets worse with every release. It is intolerably slow and buggy at this point. It used to be a fantastic piece of software, and is now a fantastic pile of shit.
Anecdotally, dynamic expressions are impossibly slow in the cases I’ve tried them.
As the author mentions, there are also a number of cases where debuggers don’t work. Personally, I’m going to reach for the tool that always works vs. sometimes works.
This is only logical if you're limited to one tool. Would you never buy a power tool because sometimes the power goes out and a hand tool is your only choice?
1. stop the program
2. edit it to add the new log
3. rebuild the program
4. run it
5. get the program to the same state to trigger the log
And then you see the result of what you printed, figure out you need something else as well, and repeat. Instead you can just trigger a breakpoint and inspect the entire program's state.
There is an inherent tradeoff between interaction and reproducibility. I think the whole conversation of debugger vs print debugging is dumb. Just do whatever makes you the most productive. Often times it is immediately obvious which makes more sense.
Spread the good word!
Is this not taught anymore? I started on borland C (the blue one, dos interface) and debugging was in the curriculum, 25+ years ago. Then moving to visual studio felt natural with the same concepts, even the same shortcuts mostly.
- run my tests, running any previous failures first - drop into the debugger when a test fails (or of course, hits a breakpoint) - end the test run after the first failure
use debugger to figure out what went wrong, fix it, wash, rinse, repeat
Setting up a debugger is the very first thing i do when i start working with a new language, and always use it to explore the code on new projects.
Being able to ask questions about the parts that are unclear (or just plain wrong) is so much easier than trying to cram the entire thing into my brain RAM.
In my experience it actually helps me learn faster too, since I rarely get stumped on random gotcha's anymore.
Not a second of that was a waste because xdebug (like all debuggers) is just that amazing.
As far as I'm concerned, breakpoints and backtraces, especially of crashes, are the superpower of debuggers. Where they are not immediately applicable, I usually don't bother.
Print debugging optimizes for timing detail and fidelity.
Debuggers optimize for data detail, at the expense of timing detail and fidelity.
In my opinion - timing & order of events is actually more meaningful.
I often don't need the extra data detail, and the debugger is going to change timing in meaningful ways.
Both are skills - both have their place. Know when logs are useful, know when debuggers are useful. Don't get stuck in the fallacy of "The one true way!" - that's the only thing I can promise is bullshit.
If you've never debugged a problem that goes away when a debugger is attached - you're woefully unprepared to have this conversation (It's much rarer, but I can even talk about times where adding a single print line was enough to change timing and make the issue harder to reproduce).
At the very least - with prints you get a tangible record of the order of operations during a reproduction. It doesn't go away, I don't have to remember how many times I've hit continue, I don't have to worry about whether pausing at this breakpoint is changing the world (and it is, because other things keep chugging - the network doesn't give a fuck that you're paused at a break point).
For really hairy bugs in programs that can't be stopped (kernel/drivers/realtime, etc) logging works.
And when it doesn't, like when you can't do I/O or switching of any kind, log non-blocking to a buffer that is dumped elsewhere.
also, related. It is harder than it should be to debug the linux kernel. Just getting a symboled stack trace is ridiculously hard.
Leave us be. We know what we’re doing.
- everyday bugs, just put a breakpoint
- rare cases: add logging
By definition a rare case probably will rarely show up in my dev environment if it shows up at all, so the only way to find them is to add logging and look at the logs next time someone reports that same bug after the logging was added.
Something tells me your debugger is really hard to use, because otherwise why would you voluntarily choose to add and remove logging instead of just activating the debugger?
Like the bugs "that disappear in a debug build but happen in the production build all the time".
Lucky you lol
What I've found is that as you chew through surface level issues, at one point all that's left is messy and tricky bugs.
Still have a vivid memory of moving a JS frontend to TS and just overnight losing all the "oh shucks" frontend bugs, being left with race conditions and friends.
Not to say you can't do print debugging with that (tracing is fancy print debugging!), but I've found that a project that has a lot of easy-to-debug issues tends to be at a certain level of maturity and as times goes on you start ripping your hair out way more
I'm having the most fun I've had in ages. It's like being Sherlock Holmes, and construction worker all at once.
Print statements, debuggers, memory analyzers, power meters, tracers, tcpump - everything has a place, and the problem space helps dictate what and when.
- You can add new traces, or modify/disable existing ones at runtime without having to recompile and rerun your program.
- Once you've fixed the bug, you don't have to cleanup all the prints that you left around the codebase.
I know that there is a good reason for debugging with prints: The debugging experience of many languages suck. In that case I always use prints. But if I'm lucky to use a language with good debugging tooling (e.g Java/Kotlin + IntelliJ IDEA), there is zero chance to ever print for debugging.
The only language where I've found a debugger particularly useful for race condition debugging is go, where it's a lot easier to synthetically trigger race conditions in my experience.
Print debugging in frontend JS/TS is literally just writing the statement "debugger;" and saving the file. JS, unlike supposedly better designed languages, is designed to support hot reloading so often times just saving the file will launch me into the debugger at the line of code in question.
I used to write C++, and setting up print statements, while easier than using LLDB, is still harder than that.
I still use print debugging, but only when the debugger fails me. It's still easier to write a series of console.log()s than to set up logging breakpoints. If only there was an equivalent to "debugger;" that supported log and continue.
no it's not lol. hmr is an outrageous hack of the language. however, the fact JS can accommodate such shenanigans is really what you mean.
sorry I don't mean to be a pedantic ass. i just think it's fascinating how languages that are "poorly" designed can end up being so damn useful in the future. i think that says something about design.
And the print will 100% change the timing.
But what to do if you have a race condition in a database stored procedure? Or in a GUI rendering code? Even web applications can experience race conditions in spite of being "single-threaded", thanks to fetches and other asynchronous operations. I never heard of somebody using ICE in these cases, nor can I imagine how it could be used - please enlighten me if I'm missing something...
> You're changing the conditions that prevent accurate measurement without modification.
Yes, but if the race condition is course-enough, like it often is in above cases, adding print/logging may not change the timings enough to hide the race.
If I find myself using a debugger it’s usually one two things: - freshly written low level assembly code that isn’t working - basic userspace app crash (in C) where whipping out gdb is faster than adding prints and recompiling.
Even never needed a debugger for complex kernel drivers — just prints.
Perhaps the debugging experience in different languages and IDEs is the elephant in the room, and we are all just talking past eachother.
If the customer has their own deployment of the app (on their own server or computer), then all you have to go with, when they report a problem, are logs. Of course, you also have to have a way to obtain those logs. In such cases, it's way better for the developers to also never use debugger, because they are then forced to ensure during development that logs do contain sufficient information to pinpoint a problem.
Using a debugger also already means that you can reproduce the problem yourself, which is already half of the solution :)
I just debug release mode instead, where print debug is usually nicer than a debugger without symbols. I could fix the situation other ways, but a non-reversible debugger doesn't justify the effort for me.
When using Xcode the debugger is right there and so it is in qt creator. I’ve tried making it work in vim many times and just gave up at some point.
The environment definitely is the main selector.
Interesting. I usually find those harder to debug with a debugger. Debuggers change the timing when stepping through, making the bug disappear. Do you have a cool trick for that? (Or a mundane trick, I'm not picky.)
I am in camp where 1% on the easy side of the curve can be efficiently fixed by print statements.
If your code can be unit tested, you can twist and turn it in many ways, if it's not an integration issue.
No shade, this was my perspective until recently as well, but I disagree now.
The tipping point for me was the realisation that if I'm printing code out for debugging, I must be executing that code, and if I'm executing that code anyway, it's faster for me to click a debug point in an IDE than it is to type out a print statement.
Not only that, but the thing that I forgot to include in my log line doesn't require adding it in and re-spinning, I can just look it up when the debug point is hit.
I don't know why it took me so long to change the habit but one day it miraculously happened overnight.
Interesting. I always viewed the interface to a debugger as its greatest flaw—who wants to grapple with an interface reimplementing the internals of a language half as well when you can simply type, save, commit, and reproduce?
I'm using IntelliJ for a Java project that takes a very long time to rebuild, re-spin and re-test. For E2E tests a 10-minute turn-around time would be blazingly fast.
But because of the tooling, once I've re-spun I can connect a debugger to the JVM and click a line in IntelliJ to set a breakpoint. Combined, that takes 5 seconds.
If I need to make small changes at that point I can usually rebuild it out exactly in the debugger to see how it executes, all while paused at that spot.
i do, because it's much faster than typing, saving, and rebuilding, etc.
No. You’re wrong.
I’ll give you an example a plain vanilla ass bug that I dealt with today.
Teammate was trying to use portaudio with ALDA on one of cloud Linux machines for CI tests. Portaudio was failing to initialize with an error that it failed to find the host api.
Why did it fail? Where did it look? What actual operation failed? Who the fuck knows! With a debugger this would take approximately 30 seconds to understand exactly why it failed. Without a debugger you need to spend a whole bunch of time figuring out how a random third party library works to figure out where the fuck to even put a printf.
Printf debugging is great if it’s within systems you already know inside and out. If you deal with code that isn’t yours then debugger is more then an order of magnitude faster and more efficient.
It’s super weird how proud people are to not use tools that would save them hundreds of hours per year. Really really weird.
I use logs and printf. But printf is a tool of last resort, not first. Debugging consideration #1 is “attach debugger”.
I think the root issue is that most people on HN are Linux bash jockeys and Linux doesn’t have a good debugger. GDB/LLDB CLI are poop. Hopefully RadDebugger is good someday. RadDbg and Superluminal would go a long long way to improving the poor Linux dev environment.
So many problems can be solved with these.
And then there's some more specialized tooling depending on what you're doing that can be a huge help.
For SQL, the query planner and index hit/miss / full table scan.
And things like valgrind or similar for cache hit/miss.
Proper observability (spans/ traces) for APIs...
Knowing that the tools exist and how to use them can be the difference between software and great software.
Though system design / architecture is very important as well.
I mean, don't get me wrong, I do agree engineers should at least be aware of the existence of debuggers & profilers and what problems they can solve. It's just that not all the stuff you've said belongs in the "must know" category.
I don't think you'll need valgrind or query planning in web frontend tasks. Knowing them won't hurt though.
All of these are useful skills in your toolkit that give you a way of reasoning about programs. Sure you can plop console.logs everywhere to figure out control/program flow but when you have a much more powerful tool specifically built for this purpose, wouldn't you, as an engineer, attempt to optimize your troubleshooting process?
But I think promoting profilers is much more important than debuggers. Far too many people I know are too eager to jump on "optimization" just because some API is too slow without profiling it first.
For JS in the browser, there's a often chain of transformations - TypeScript, Babel, template compilation, a bundler, a minifier - and each of these makes the browser debugger work worse -- and it's not that great to begin with, even on plain JS.
Add that to the fact that console.log actually prints objects in a structured form that you can click through and can call functions on them from the console, and you start to see why console.log() is the default choice.
I work on maintaining a 3D rendering engine written completely in Typescript, along with using a custom, stripped down version of three.js that I rely on for primitives; and no amount of console.logging will help when you're trying to figure out exactly what's going wrong in a large rendering pipeline.
I do use console.logs heavily in my work, but the debugger and profiler are instrumental in providing seamless devex.
> TypeScript, Babel, template compilation, a bundler, a minifier
During development you have access to source maps, devtools will bind breakpoints, show original typescript code and remap call stacks across bundlers. All modern browsers support mapped debugging, also wrt profiling it can also be symbol mapped to the original sources which makes minified builds diagnosable if you ship proper source maps, which during development you ideally should.
-=-
edit: additional info;
I would also like to say console.log and debugging/profiling are not in a competition. both are useful in different contexts.
for example I will always console.log a response from an API because I like having a nice nested representation that I can click through, I'll console.log objects, classes and everything to explore them in an easier way. this is also great for devex.
I'll use the debugger when I want to pause execution at an intermediate step; for example see the result of my renderer before the postprocessing step kicks in, stop it and inspect shader code before its executed. it's pretty useful.
As mentioned originally; these are TOOLS in your toolkit, you don't have to do a either/or between them.
If you don't already know which tool to use / how to diagnose the problem, you'll instead of banging your head against the wall, you'll think - "how do i figure out this thing - what is the right tool for this job"? and then you'll probably find it, and use it, because people are awesome and build incredibly useful free / open source software.
"try stuff until it works" is so common, and the experience needed to understand how to go about solving the problem is within reach.
Like especially with llms, "what's the right tool to use to solve problem x i'm having? this is what's going on. i'm on linux/macos, using python" or w/e
It won't run if I compile with debug info. I think it's due to a 3rd party proprietary library. So, to run the app I have to use release profile, with debug info stripped.
So, when I fire up gdb, I can't see any function information or anything, and it has so many system calls it's really difficult to follow through blindly.
So, what is the best way to handle this?
Of course that may require digging down pretty low, which is difficult in itself.
Edit: also there's split-debuginfo which puts debug info in separate file. It could help if the reason you can't run it is the debug info itself. Which feels unlikely, but :shrug:.
Curious if it's possible could it be because of protobuf implementation, which is used between UI and the server, and my error is occurring on the UI side.
So, after reading a bit, this is what I find
>Deterministic serialization is not canonical. The serializer can generate different output for many reasons, including but not limited to the following variations:
> The binary is built with different flags (eg. opt vs. debug).
My knowledge on this is pretty limited, so I could be wrong. But, this could be a reason. Maybe someone more knowledgeable on this matter can shade some lights. And I should've studied more on this before ... heh.
[profile.release]
debug = true
Not related to OP, but debugging is often about finding where an invariant is broken, so it feels like using LLM to navigate a debugging loop may be useful as it's not a complicated but repetitive task. However in the morning I struggle to imagine how to do that.
I agree that adhoc dynamic expression evaluation at run time is very useful and can only be done in a debugger.
Shame as that's likely the only option with significant universal UX advantage vs. sprinkling prints...
I once wrote a program that opened up all of my code, and at every single code curly brace, it added a macro call, and a guid.
void main() { DEBUGVIKINGCODER("72111b10c07b4a959510562a295cb2ac");
...
}
The DEBUGVIKINGCODER macro, or whatever I called it, was a no-op in release. But in Debug or testing builds, would do something like:
DebugVikingCoder coder##__LINE__("72111b10c07b4a959510562a295cb2ac");
The constructor for DebugVikingCoder used a thread-local variable to write to a file (named after the thread id). It would write, essentially,
Enter 72111b10c07b4a959510562a295cb2ac (epoch time)
Exit 72111b10c07b4a959510562a295cb2ac (epoch time)
Then I wrote another program that would read those all up, and would also read the code, and learn the File Name, Line Number of every GUID...
And, in Visual Studio, this tool program would print to the Output window, the File Name and Line Number, of every call and return.
And, in Visual Studio, you can step forward AND BACK in this Output window, and if you format it correctly, it'll open the file at that point, too.
So I could step forwards and backwards, through the code, to see who called where, etc. I could search in this Output window to jump to the function call I was looking for, and then walk backwards...
Then I added some code that would compare one run to another, and argued we could use that to figure out which of our automated tests formed a "basis set" to execute all of our code...
And to recommend which automated tests we should run, based on past analysis.
In addition to being able to time calls to functions, of course.
So then I added printing out some variables... And printing out lines in the middle of functions, when I wanted to time a section...
And if people respected the GUIDs, making a new one when they forked code, and leaving it alone if they moved code, we could have tracked how unit tests and other automation changed over time.
That got me really wishing that every new call scope really did have a GUID, in all the code we write... And I wished that it was essentially hidden from the developers, because who wants to see that? But, wow, it'd be nice if it was there.
I know there are debuggers that can go backwards and forwards in time... But I feel like being able to compare runs, over weeks and months, as the code is changing, is an under-appreciated objective.
And slower of course, they are not free.
Some other things I'd add:
Some debuggers allow you to add actions. For example logging at the breakpoint is great if I can't modify the source, plus there's nothing to revert afterward. This just scratches the surface. Some debuggers allow you to see entire GPU workloads, view textures etc.
Debuggers are extremely useful for exploring and helping edit code. I can't be the only person that sprinkles breakpoints during development which helps me visualise code flow and quickly jump between source locations.
They're not just for debugging.
This isn't always the case. Maybe it's really hard in a lot of cases, but it's always not impossible.
A bit of counterpoint here
I’m not saying you’re wrong or I’m right, just that we have diametric opposite opinions on this.
Readable code, though, is written with the reading view in mind.
Maybe 0.1%
There are a few other instances where the interaction offers notable benefits—bugs in the compiler, debugging assembly, access to registers, a half-completed runtime or standard library that occludes access to state so that you might print it. If you have the misfortune of working with C or C++, you have the benefit of breaking on memory access—but I tend to file this in the "half-completed runtime" category. There are also a few "heisenbugs" that may actually prevent the bug from occurring by using print itself; but I've only run into this I think twice. This is also possible with the debugger, but I've only run into that once. The only way out of that mess is careful reasoning, and i recommend printing the code out and using a pen.
I also strongly suspect that preference for print debugging vs interactive debuggers comes down to internal conception of the runtime and aesthetic preference. I abhor debuggers—especially thosr in IDEs. I think they tend to reimplement the runtime of a language a second time, except with more bugs and a less intuitive interface. But I have the wherewithal to realize that this is ultimately a preference.
The fact that there ist No Debugger is super unfortunate
It seems to me that this is one of the many phenomena where people want to judge and belittle their peers over something completely trivial.
Personally, I get the appeal of printing out debugging information, especially if some bug is rare and happens in unpredictable times (such as when you are sleeping). But the amount of info you get this way is necessarily lower than what can be gleaned from a debugger.
Alot of programmers work in a Linux environment.
It seems like windows, ide and languages are all pretty nicely integrated together?
Not only, and not really. After all, for all its warts Visual Studio is still a decent debugger for C/C++. IntelliJ has pretty good debuggers across all of their IDEs for almost all languages (including things like automatically downloading and/or decompiling sources when you step into external libraries).
Even browsers ship with built-in debuggers (and Chrome's is really good). I still see a lot of people (including my colleagues) often spend inordinate amounts of time console.log'ing when just stepping though the program would suffice.
I think it's the question of culture: people are so used to using subpar tools, they can't even imagine what a good one may look like. And these tools constantly evolve. Here's RAD Debugger by Ryan Fleury: https://threadreaderapp.com/thread/1920345634026238106.html
1. Code where the granularity of state change is smaller than a function call. Sometimes you actually have to step through things one instruction at a time, and I'm lucky enough to have such problems to solve. You can't debug your assembly with printf(), basically[1a].
2. State changes that can't be easily isolated. Sometimes you want to log when something change but can't for the life of you figure out when it's changing. Debuggers have watchpoints.
But... that's really it. If I'm not hitting one of those I'm not reaching for the debugger. Logging is just faster, because you type it in right at the code you're already reading.
[1a] Though there's a caveat: sometimes you need to write assembly and don't even have anything like a printk. Bootstrap code for a new device is a blast. You just try stuff like writing one byte to a UART address or setting one GPIO pin as the first instructions and hope it works, then use that one bit of output to pull the rest up.
But I spend far more time reading and thinking than I do typing. Input mechanics just aren't the limiting factor here.
Secondly, in your example, no need to label the names. This is almost always understood by context. So, pretty manageable. e.g. in JS: log(`${longvarname}, ${secondvarname}`)
There are a couple of ways to resolve this conundrum, and you seem to be locked on the less likely one.
What if… that weren’t an essential skill?
The debugger will never be completely transparent, it also eats resources in parallel to your application, and peeking into the session also introduces timing issues, short of the debugger itself having its own bugs.
I'm saying it would be dumb to dismiss all other tools for the love of debuggers, it's just one tool in the toolbox.