Production debugging: Is there a less intrusive way than WinDbg? - debugging

I was wondering if there is a less intrusive way to analyze a running, managed process in production environments.
Less intrusive meaning:
No delay of execution when attaching the debugger.
No delay of execution when getting basic stats like running threads.
In the Java world there is a such a tool part of the JDK. I was wondering if there're similar tools in the .NET world.
The tool should answer questions like:
What are the thread pool parameters? Same as "!threadpool" in Windbg.
What are the callstacks of my currently running threads (yep, you get it from the Java tool :) ).
Basic heap analysis e.g. howmany objects of type ABC.
Any ideas?
Alex

If I understand you correctly, you don't want to actually debug the program, only get some basic information. In such cases, Process Explorer may be sufficient.

As Oefe says, you can get a lot of info including the stacks of all threads from Process Explorer. Also, the .NET runtime has a number of useful performance counters, that may give you some insight. If you have special needs, your application can publish its own counters.

Here is production debugging in a non-intrusive manner using ETW and another one

It depends on what you want to debug. WinDbg is the giant hammer of Windows debugging, suitable for debugging anything from kernel extensions on up.
If you just want to debug a program, most people just use visual studio, which will attach to a running processs.
However, #oefe may have the bull by the horns here. When most people say 'debugger' they want backtraces and breakpoints and such. In Java, you need to make prior arrangements to attach that sort of debugger. Either Windbg or visual studio (-debugexe) is more convenient than that.

Related

Is there a way to debug console programs from the command-prompt?

At the moment, I’m writing a small console program. It is small enough that I have not bothered to use an IDE, so I’m just using Notepad2 and cl.exe to edit and build the program, and the command-prompt to run and test it.
This works just fine for the most part (don’t freak out, but I’m actually using a bit of “printf debugging” here and there). Unfortunately certain problems are not quite as easy to fix. For example, a pointer/array problem can cause the program to crash.
I know about—and have—Visual Studio, Ollydbg, Windbg, etc. but those are all fancy, schmancy graphical IDEs and debuggers that are too cumbersome to fire up for what usually amounts to a five-second check before going back to the editor. My edit-compile-test cycle is often as low as 10-15 seconds, so such debuggers become an unacceptable bottleneck.
I’m trying to find out if there is a fast and easy method to debug a console application from the command-prompt. Is there a modern text-mode debugger? (Obviously debug isn’t going to be of much use.) Is there some other way—TSR‽—to view values, catch exceptions, or set breakpoints and step through a program?
You need ntsd (part of the same package as WinDbg, shares all comands) - http://msdn.microsoft.com/en-us/windows/hardware/gg463009
Also using free Visual Studio Express may make search for debugging advices easier...

Windows Debugger Road-Map

It seems there are dozens of debugger and debugging tools that Microsoft produces, which creates a maze of choices and questions concerning which tool to apply, and when. For example, there is windbg - and the debugger built into Visual Studio. Both can access minidumps. Why would I choose one over the other?
Dr. Watson was the default post-mortem crash analysis tool of the past. It has now been replaced with "Problem Reports & Solutions". Which is in turn replaced with IIS Exception Monitor on servers? And perhaps all of this is built on top of "Microsoft CDB Debugger," or perhaps that is a another duplicate tool? ADPlus, yet another one, is built on CDB Debugger. The maze seems to go on endlessly.
Can someone provide a link to a taxonomy or roadmap of all these tools, with comments of which are being deprecated (Dr. Watson?) and what "tool direction" debug students should absorb? I'm sure there are a number of tools and base libraries I've not mentioned here. it would be nice to know the dependencies between them too (such as ADPlus using the CDB Debugger).
I've found this link to be helpful, since it answers some of the questions I'm asking - though the material is dated. Any other resources that give a similarly simple compare / contrast run-down?
There is no difference between CDB and NTSD, other than how they spawn new windows. Choosing when to use Visual Studio over the command line debuggers is sometimes a matter of personal choice, but sometimes the command line is a better tool for the job. Once you get good at using the command line debuggers, you can get things done much more quickly. I suspect there are a few scenarios that remain where you can only debug a specific problem with the command line debugger, but I can't think of any off hand. The third debugger you've missed is kd, which is the kernal debugger. If you want to debug kernal mode stuff (i.e. your device drivers you've written) it's really your only choice.
CDB, NTSD and KD are all part of the debugging tools for Windows, itself part of the DDK. Visual Studio does not depend on the other debugging package and vice-versa.
Watson and the like are not debuggers. They merely observe and report. I suspect the best advice there is use whichever one is appropriate to your problem. I mean, there are lots of tools for all sorts of different MS technologies. E.g. Orca for MSI databases. All of these products are unrelated, often released and maintained by different divisions, etc. As a result, I doubt you'll find a chart showing their relationships since they are so diverse.

Are there any WinDBG replacements with a better GUI?

I'm using WinDBG occasionally to analyze problems in production environment, where VS cannot be installed. There's no doubt it's an extremely powerful tool, but using it is a bit annoying. Even though the product is frequently updated, its GUI goes back to the Win95 days or so, and its usability is accordingly. Having to fight the GUI to layout the windows the way I want, and having to remember all those textual commands, is just quite low a standard for a modern desktop application.
AFAIK, WinDBG is pretty much built on top of CDB, which is a command line debugger. Being so, it shouldn't be that hard to built a modern days GUI wrapper that will replace the existing dinosaur. Has anyone ever done that? Am I the only one having those mixed feelings toward WinDBG?
(BTW, I know I can create a dump and take it back to where I have VS, but I sometimes have to debug 64 bit processes, and I don't have a 64 bit dev machine. Sad, but true)
Consider the new WinDbg. (It's still in Preview). It also supports Time Travel Debugging.
You can install it from the Microsoft Store, or use the links here.
Here is what's new with Windbg Preview.
Have look at this if you fancy trying out a GUI to replace WinDbg.
EDIT:
Since SOS Assist is no longer available, this answer should be deleted. As this answer has been accepted, I personally cannot delete it. In that, please ignore my answer.
I guess thats too much to expect. With such a large number of commands that it has, it will not be trivial to have UI that displays everything in fancy controls. It might also make it bulkier, slower.
However it does provide you with controls that any user mode application debugger should have. It displays most frequently needed information like call stack, local variables, threads and so on in seperate windows.
But if you need more advanced debugging feature, you alwalys have the command interface.
WinDBG is pretty much it, no one has ever bothered to write their own UI for it. Even with its quirks I'm a fan because it's mostly command line driven. So, to each their own :)
Note that the VS 2011 Dev Preview basically integrates WinDBG support, so in the future VS will be the new WinDBG UI.
-scott

What rarely used debugging tools you found useful?

What rarely used debugging tools you found useful ?
My recent debugging situation on Visual Studio required trapping the breakpoint on fresh built 32-bit DLL, which was loaded by GUI-less executable, which was spawned by COM+ server on remote x64 machine, which was called through RPC from actual GUI. As usual, all worked well on all 32 bit machine, but kept failing on "machine other than development one". So remote debugging was inevitable.
So after scratching the head beaten against wall for 2 days, I have added 10 sec delay into DLL attach entry point and used Microsoft Remote Debugger wich I never used before. It saved my day.
Another favorite: Java JMX console as a performance "debugging" tool. You can see all threads, memory chart, have a snapshot of any thread stack any time you click. Clicking several times helps to find what exactly is slow in J2EE application.
Process Monitor and other Mark Russinovich's tools.
A logic analyzer plugged to CPU pins and able to disassemble executed code. I tracked a bug in the boot sequence of an embedded system.
I find printf to be the most useful.
These - in my experience at least - do not seem to be the intuitive first choice for many when debugging apps accessing a database (i.e. the majority), that perhaps they should be :
SQL Profiler (SQL Server)
TKPROF (Oracle)
Another interesting combination was using eclipse running in a virtual machine, accessing a remote server, attaching to the Tomcat process there; and doing it from two different machines to debug two different packages simultaneously.
All time favorite is depends.exe, for finding out why a dll or exe is not starting http://dependencywalker.com/
For performance, at my former job we used to have really simple to use C++ macro's that did statistics on runtime function calls. This is so much better than a profiler, because you can use it from your regular IDE, and it allows you to zoom in on the code you are optimizing.
In my new job, I wrote a C# version of the same idea.
WinDbg and other lower level debuggers are the ultimate weapon if you know the tricks and tips.
For Windows/.Net development I am always using Debugview and Ildasm.

Why is debugging better in an IDE? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
I've been a software developer for over twenty years, programming in C, Perl, SQL, Java, PHP, JavaScript, and recently Python. I've never had a problem I could not debug using some careful thought, and well-placed debugging print statements.
I respect that many people say that my techniques are primitive, and using a real debugger in an IDE is much better. Yet from my observation, IDE users don't appear to debug faster or more successfully than I can, using my stone knives and bear skins. I'm sincerely open to learning the right tools, I've just never been shown a compelling advantage to using visual debuggers.
Moreover, I have never read a tutorial or book that showed how to debug effectively using an IDE, beyond the basics of how to set breakpoints and display the contents of variables.
What am I missing? What makes IDE debugging tools so much more effective than thoughtful use of diagnostic print statements?
Can you suggest resources (tutorials, books, screencasts) that show the finer techniques of IDE debugging?
Sweet answers! Thanks much to everyone for taking the time. Very illuminating. I voted up many, and voted none down.
Some notable points:
Debuggers can help me do ad hoc inspection or alteration of variables, code, or any other aspect of the runtime environment, whereas manual debugging requires me to stop, edit, and re-execute the application (possibly requiring recompilation).
Debuggers can attach to a running process or use a crash dump, whereas with manual debugging, "steps to reproduce" a defect are necessary.
Debuggers can display complex data structures, multi-threaded environments, or full runtime stacks easily and in a more readable manner.
Debuggers offer many ways to reduce the time and repetitive work to do almost any debugging tasks.
Visual debuggers and console debuggers are both useful, and have many features in common.
A visual debugger integrated into an IDE also gives you convenient access to smart editing and all the other features of the IDE, in a single integrated development environment (hence the name).
Some examples of some abilities that an IDE debugger will give you over trace messages in code:
View the call stack at any point in time, giving you a context for your current stack frame.
Step into libraries that you are not able to re-compile for the purposes of adding traces (assuming you have access to the debug symbols)
Change variable values while the program is running
Edit and continue - the ability to change code while it is running and immediately see the results of the change
Be able to watch variables, seeing when they change
Be able to skip or repeat sections of code, to see how the code will perform. This allows you to test out theoretical changes before making them.
Examine memory contents in real-time
Alert you when certain exceptions are thrown, even if they are handled by the application.
Conditional breakpointing; stopping the application only in exceptional circumstances to allow you to analyse the stack and variables.
View the thread context in multi-threaded applications, which can be difficult to achieve with tracing (as the traces from different threads will be interleaved in the output).
In summary, print statements are (generally) static and you'll need to re-compile to get additional information if your original statements weren't detailed enough. The IDE removes this static barrier, giving you a dynamic toolkit at your fingertips.
When I first started coding, I couldn't understand what the big deal with debuggers was and I thought I could achieve anything with tracing (granted, that was on unix and the debugger was GDB). But once you learn how to properly use a graphical debugger, you don't want to go back to print statements.
An IDE debugger lets you change the
values of variables at run-time.
An IDE
debugger lets you see the value of
variables you didn't know you wanted
to see when execution began.
An IDE
debugger lets you see the call stack
and examine the state of the
function passed weird values.
(think this function is called from
hundreds of places, you don't know
where these weird values are coming
from)
An IDE debugger lets you
conditionally break execution at any
point in code, based on a condition,
not a line number.
An IDE debugger will let you examine the state of the program in the case of an unhandled exception instead of just crapping out.
Here's one thing that you definitely cannot debug with "print" statement, which is when a customer brings you memory dump and says "your program crashed, can you tell me why?"
Print statements all through your code reduces readability.
Adding and removing them for debug purposes only is time consuming
Debuggers track the call stack making it easy to see where you are
Variables can be modified on the fly
Adhoc commands can be executed during a pause in execution to assist diagnosing
Can be used IN CONJUNCTION with print statements : Debug.Write("...")
I think debugging using print statements is a lost art, and very important for every developer to learn. Once you know how to do that, certain classes of bugs become much easier to debug that way than through an IDE. Programmers who know this technique also have a really good feel of what's useful information to put in a log message (not to mention you'll actually end up reading the log) for non-debugging purposes as well.
That said, you really should know how to use the step-through debugger, since for a different class of bugs it is WAY easier. I'll leave it up to the other excellent answers already posted to explain why :)
Off the top of my head:
Debugging complex objects - Debuggers allow you to step deep into an object's innards. If your object has, say, an array of array of complex objects, print statements will only get you so far.
The ability to step past code - Debuggers will also allow you to skip past code you don't want to execute. True, you could do this manually as well, but it's that much more code you have to inject.
As alternative to debug in IDE you can try great Google Chrome extension PHP Console with php library that allows to:
See errors & exception in Chrome JavaScript console & in notification popups.
Dump any type variable.
Execute PHP code remotely.
Protect access by password.
Group console logs by request.
Jump to error file:line in your text editor.
Copy error/debug data to clipboard (for testers).
I haven't been developing for nearly 20 years, but I find that using a IDE / debugger I can :
see all kinds of things I might not have thought to have included in a print statement
step through code to see if it matches the path I thought it would take
set variables to certain values to make code take certain branches
One reason to use the IDE might be that modern IDEs support more than simple breakpoints. For example, Visual Studio offers the following advanced debugging features:
define conditional breakpoints (break only if a condition is met, or only on the n-th time the statement at the breakpoint is executed)
break on an unhandled exception or whenever a (specific) ecxeption is to be thrown
change variable while debugging
repeating a piece of code by setting the next line to be executed
etc.
Also, when using the debugger, you won't have to remove all your print statements once you have finished debugging.
In my experience, simple printouts have one huge advantage that no one seems to mention.
The problem with an IDE debugger is that everything happens at real time. You halt the program at a certain time, then you step through the steps one at a time and it is impossible to go back if you suddenly want to see what happened before. This is completley at odds with how our brain works. The brain collects information, and gradually forms an oppinion. It might be necessary to iterate the events several times in doing so, but once you have stepped past a certain point, you cannot go back.
In contrast to this, a selected series of printouts/logging gives you a "spatial projection of the temporal events". It gives you a complete story of what happened, and you can go back and fourth several times very easily by just scrolling up and down. It makes it easy to answer questions like "did A occur before B happened". It can make you see patterns you wernt even looking for.
So in my experience. IDE and debuggers are fantastic tools to solve simple problems when something in one single call-stack went wrong, and explore the current state of the machine at a certain crash.
However, when we approach more difficoult problems where gradual changing of state is involved. Where for example one algorithm corrupted a data structure, that in turn caused anohter algorithm to fail. Or if we want to answer questions like "how often do this happen", "do things happen in the order and in the way as I imagine them to happen". etc. Then the "old fashined" logging/printout technique has a clear advantage.
The best things is to use either technique when it is most suitable, for example use logging/printouts to get to some bugs, and pause at a breakpoint where we need to explore the current state more in detail.
There are also hybrid approaches. For example, when you do console.log(object) you get a data-structure widget in the log that you can expand and explore more in detail.This is many times a clear advantage over a "dead" text log.
One thing that I'm surprised I haven't seen in another answer is that the 2 debugging methods are not mutually exclusive.
printf debugging can work quite nicely even if you're using a standard debugger (whether IDE based or not). In particular with a logging framework so you can leave all or most of in the released product to help with diagnosing customer problems.
As noted in pretty much all the other answers here, the key nice thing about a standard debugger is that it allows you to more easily examine (and potentially change) the details of the program state. You don't have to know up front what you might want to look at - it's all available at your fingertips (more or less).
Because debugging multi-threaded applications with print statements will drive you bananas. Yes you can still do it with print statements but you'd need a lot of them and unravelling the sequential print out of statements to emulate the multi-threaded executiong would take a long long time.
Human brains are only single-threaded unfortunately.
Since you asked for pointers to books... As far as Windows debugging goes, John Robbins has several editions of a good book on Windows debugging:
Debugging Applications for Microsoft .NET and Microsoft Windows
Note that the most recent edition (Debugging Microsoft .NET 2.0 Applications) is .NET only, so you might want an older one (like in the first link) if you want native code debugging (it covers both .NET and native).
I personally feel the answer is as simple as "A integrated debugger/IDE gives you a wealth of different information quickly without the need for punching in commands. The information tends to be there in front of you without you haven't tell it what to show you.
The ease in which the information can be retrieved is what makes them better than just command-line debugging, or "printf" debugging.
Advantages of a debugger over a printf (note not an IDE debugger but any debugger)
Can set watchpoints.
This is one of my favourite ways of finding memory corruptions
Can debug a binary that you can't recompile at the moment
Can debug a binary that takes a long time to recompile
Can change variables on the fly
Can call functions on the fly
Doesn't have the problem where debug statemenets are not flushed and hence timing issue can not be debugged acuratly
Debuggers help with core dumps, print statements dont'
This is what I use most on VS.NET debugging windows:
Call stack, which is also a great way to figure out someone else's code
Locals & Watches.
Immediate window, which is basically a C# console and also lets me change variable contents, initialize stuff etc.
The ability to skip a line, set the next statement to be executed somewhere else.
The ability to hover over variables and have a tool-tip showing me their values.
In summary, it gives me a 360 degree view of the state of my executing code, not just a small window.
Never found a book teaching this kind of stuff, but then again, it seems to be quite simple, it's pretty much WYSIWYG.
A debugger can attach to a running process
Often easier to debug threaded code from a debugger
With an IDE debugger you can see the values of ALL the variables in the current scope (all the way up the call stack) whenever you halt execution.
Print statements can be great but dumping so much information to the screen at any given place can produce a whole lot of print statements.
Also, many IDE debuggers let you type in and evaluate methods, and evaluate members while you are halted, which further increases the amount of print statements you'd have to do.
I do feel that debuggers are better for some languages than for others however...
My general opinion is that IDE debuggers are absolutely, amazingly wonderful for managed languages like Java or C#, are fairly useful for C++, and are not very useful for scripting languages like Python (but it could be that I just haven't tried a good debugger for any scripting languages yet).
I absolutely love the debugger in IntelliJ IDEA when I do Java development. I just use print statements when I use Python.
As someone said above: Debugger != IDE.
gdb and (back in the day) TurboDebugger (stand-alone) work just fine for the languages they support[ed], thank you. (or an even older technology: Clipper debugger linked into the xBase executable itself) -- none of these required an IDE
Also, though C/++ coding is more rare, printf statements sometimes mask off the very bug you are trying to find! (initialization problems in auto vars on the stack, for instance, or memory allocation/alignment)
Finally, as others stated, you can use both. Some real-time-ish problems almost require a print, or at least a judicious "*video_dbg = ( is_good ? '+' : '-');" somewhere into video memory. My age is showing, this was under DOS :-)
TMTOWTDI
In addition to much of what the other posters have said, I really like stepping through one line at a time along with the computer, as it forces me to think about one line at a time. Often I will catch the bug without even looking at variable values simply because I am forced to look at it as I click the 'next line' button. However, I don't think my answer will help you, Bill, because you probably have this skill already.
As far as learning resources go, I haven't used any -- I just explore all the menus and options.
Is this even real question from real programmer?
Anyone who spent even 5 mins debugging with print statements and debugging with IDE - it will OCCUR to him/her without even asking!
I've used both prints and IDEs for debugging and I would much rather debug using an IDE. The only time for me when that doesn't work is in time critical situations (like debugging online games) where you litter the code with print statements and then look at the log files after it has gone horribly wrong. Then if you still cannot figure it out, add more prints and repeat.
Just wanted to mention a useful feature of a console debugger vs printf and vs debugger in an IDE.
You can attach to a remote application (obvioustly, compiled in DEBUG mode) and inspect its state dumping the debugger output to a file using POSIX tee utility. Compared to printf, you can choose where to output the state in run-time.
It helped me a lot when I was debugging Adobe Flash applications deployed in an agressive environment. You just need to define some actions that print required state in each breakpoint, start the console debugger with fdb | tee output.log, and walk through some breakpoints. After that you can print the log and analyse the information by thorough comparison of the state in different breakpoints.
Unfortunatelly, this feature [logging to a file] is rarely available in GUI debuggers, making developers compare the state of objects in their head.
By the way, my opinion is that one should plan where and what to debug before staring a debugger.
Well another thing is that if you join a new old project and nobody really knows how the code is doing what it's doing, then you can't debug by echoing variables/objects/... b/c you have no idea what code is executed at all.
At my job I am facing exactly that kind of situation and visual XDebuging helps me getting an idea about what is going on and where, at all.
Best regards
Raffael
In addition to the many things that have been already mentioned, one of the most important advantages of a debugger over printf is that using printf statements assumes that you know in which function the bug resides. In many cases you don't, so you have to make a few guesses and add print statements to many other functions in order to localise it. The bug may be in framework code or somewhere far removed from where you think it is. In a debugger it is far easier to set breakpoints to examine the state in different areas of the code and at different points in time.
Also, a decent debugger will let you do printf-style debugging by attaching conditions and actions to breakpoints, so that you still retain the benefits of printf debugging, but without modifying the code.
Debugging in an IDE is invaluable in an environment where error logs and shell access are unavailable, such as a shared host. In that case, an IDE with a remote debugger is the only tool which allows you to do simple things such as view stderr or stdout.
A problem with using print statements is it makes a mess of your code. IE, you have a function with 10 parts to it and you know it crashes somewhere, but you're not sure where. So you add in 10 extra print statements to pinpoint where the bug is. Once you've found and solved your bug, you now have to clean up by removing all of those print statements. Maybe you'll do that. Maybe you'll forget and it'll end up in production and your user's console will be full of debug prints.
Wauw, do I like this question. I never dared to pose it...
It seems that people just have different ways of working.
For me what works best is:
Having a solid mind model of my code, including memory management
Using instrumentation (like print statements) to follow what's happening.
I've earned my living programming for over 40 years now, working at non-trivial technical and scientific applications in C++ and Python daily, and I have the personal experience that a debugger doesn't help me a bit.
I don't say that's good. I don't say that's bad. I just want to share it.
It's not just debugging. An IDE helps you build better software faster in a lot of ways:
refactoring tools
intellisense to make api's more discoverable, or remind of exact spelling/case of familiar items(not much use if you've used the same system for 15 years, but that's rare)
save on typing by autocompleting variable and class names
find certain kinds of errors before you even start to compile
Automatically jump to variable/method/class declarations/definitions, even if they're not in the same file or folder.
Break on unhandled and handled exceptions
I could go on.

Resources