It seems there are dozens of debugger and debugging tools that Microsoft produces, which creates a maze of choices and questions concerning which tool to apply, and when. For example, there is windbg - and the debugger built into Visual Studio. Both can access minidumps. Why would I choose one over the other?
Dr. Watson was the default post-mortem crash analysis tool of the past. It has now been replaced with "Problem Reports & Solutions". Which is in turn replaced with IIS Exception Monitor on servers? And perhaps all of this is built on top of "Microsoft CDB Debugger," or perhaps that is a another duplicate tool? ADPlus, yet another one, is built on CDB Debugger. The maze seems to go on endlessly.
Can someone provide a link to a taxonomy or roadmap of all these tools, with comments of which are being deprecated (Dr. Watson?) and what "tool direction" debug students should absorb? I'm sure there are a number of tools and base libraries I've not mentioned here. it would be nice to know the dependencies between them too (such as ADPlus using the CDB Debugger).
I've found this link to be helpful, since it answers some of the questions I'm asking - though the material is dated. Any other resources that give a similarly simple compare / contrast run-down?
There is no difference between CDB and NTSD, other than how they spawn new windows. Choosing when to use Visual Studio over the command line debuggers is sometimes a matter of personal choice, but sometimes the command line is a better tool for the job. Once you get good at using the command line debuggers, you can get things done much more quickly. I suspect there are a few scenarios that remain where you can only debug a specific problem with the command line debugger, but I can't think of any off hand. The third debugger you've missed is kd, which is the kernal debugger. If you want to debug kernal mode stuff (i.e. your device drivers you've written) it's really your only choice.
CDB, NTSD and KD are all part of the debugging tools for Windows, itself part of the DDK. Visual Studio does not depend on the other debugging package and vice-versa.
Watson and the like are not debuggers. They merely observe and report. I suspect the best advice there is use whichever one is appropriate to your problem. I mean, there are lots of tools for all sorts of different MS technologies. E.g. Orca for MSI databases. All of these products are unrelated, often released and maintained by different divisions, etc. As a result, I doubt you'll find a chart showing their relationships since they are so diverse.
Related
I've spent the last few days searching google, blogs and MSDN looking for any small scrap of info on how "interop" or "mixed mode" debugging is implemented in Visual Studio.
I'm attempting to implement my own debugger for a custom VM (actually, it gets JIT-ed to native code, which means I'll have to rebuild callstacks for the normal native debugger whenever it enter's the JIT-ed code), but I can't find any info on how you can actually interact with VS's native debugger and do the fancy "native/managed" like transitions that VS's mixed mode debugging pulls off.
So far I've found a few things that provide useful tidbits, but not enough to actually interact with the debugger. The best, most useful articles I've found are:
Mike Stall's various blog posts (mainly these two) while providing useful info and glimpses at the inner workings, they seem to be very heavily tied to .Net debugging.
MSDN's Creating a Basic Debugger gives a nice overview of how MS implements it's debuggers and how you can implement one on the same tech stack. Unfortunately, this doesn't really provide any real info on how to pass data off to VS's native debugger. (Combined with Mike Stall's blogs, it seems like both debuggers would be waiting for the same events, so how do you actually splice the results together in a meaningful way?)
The Debugger Engine API (aka DbgEng) documentation. I found this by running Dependency Walker on VS's native debug engine (NatDbgDE.dll, it seems to export the DebugCreate function which I could only find in this documentation, so it could be related). Unfortunately I've had very little success in even doing any debugging through this and it seems to be poorly documented, although the documentation does seem to indicate this is what I'll probably want to deal with in the end. It also doesn't really say how I can work cooperatively with another debugger and makes no mention of VS's debugger, so I could be going down the wrong path entirely with this anyway.
How can I get started with writing a debugger that will work cooperatively with VS's native debugger?
Thanks!
I'm afraid there's not much documentation on this subject. The resources you mention are all pretty old at this point. My suggestion is to integrate with Visual Studio's newer debug engine (Concord). Concord is used as the debug engine for native debugging from Visual Studio 2012 forward and for all debugging in Visual Studio 2013 forward. One of the design goals of Concord was to simplify mixed mode debugging. It is also designed to be easily extensible.
I created some Concord extensibility documentation that may be of help. It is mainly focused on expression evaluators, but it has some good information for getting started with Concord. Another resource that may be of use is the source code for the Concord-based debugger in Python Tools for Visual Studio. This is a full implementation of a mixed mode debugger that integrates with Concord and allows mixed mode debugging with Python.
As part of the work I've been doing to answer this question about the technical workings of a glitch in Pokémon Red, I've been looking for a way to use a standard debugger to debug a Game Boy ROM. Although many of the emulators I've found have some support for debugging, nothing I've found so far has been helpful.
As a background, as of now I have tried to use the Visual Boy Advance built-in features to do debugging, but they aren't particularly useful for what I'm trying to do. VBA lacks the ability to set breakpoints, and since it steps forward at the level of frames rather than instructions I'm unable to see how the code is executing when I actually need it to. Although VBA says that it supports GDB debugging, I have been completely unable to get it working. I tried cross-compiling GDB for ARM as per the instructions, but could not get GDB to connect to the emulator (it would recognize that there was a program to connect to, but reported that the protocol had been violated). I repeated this with similar success in both Windows with Cygwin and on Ubuntu Linux. A friend and I tried to use Insight/GDB, but ran into exactly the same problems.
I also tried to use the NO$GBA debugger, but it refused to load my ROM for Pokémon Red (and then insulted me by saying that nothing I could try to do would fix it, as the file was just flat-out wrong).
Additionally, I tried downloading this version of Visual Boy Advance that claims to have a debugger in it, but for some reason I can't get it to enable the debugger. Pressing F11 as per its instructions has no effect whatsoever.
I believe that I've done my due diligence trying to get a debugger working, and I'm surprised that not a single one of them has worked. Does anyone know of a simple, straightforward way to debug Game Boy games using standard debugging techniques? I'm interested mostly in being able to put in memory write breakpoints (to see what routine is clobbering certain parts of memory). I would really appreciate it if someone with first-hand experience doing this could provide details on how to do this, as online resources on the subject seem pretty limited.
If you just want to debug your old gameboy games you can also use bgb which has several debugging options such as tracing, breakpoints, profiler and a lot more.
No$GBA is for GBA games; you want NO$GMB. Note that it's very buggy, and without a registered version (which may be impossible to get legitimately) rather crippled.
bgb is free and is very similar to No$GMB, but even buggier.
VBA is supposed to have a debugger, but there are a million different versions out there, so good luck finding the right one.
Check out the site GbaDev.org and look on the forums. This is the best spot on the web for GBA or even GBC questions. I can tell you that there are many versions of VBA and no$ out and about. The No$ you want was technically a pay for version, but Martin Korth hasn't been answering emails or anything for years now and I'm not sure of its status anymore. I can also answer some questions for you personally if you'd like or help you with the debugger.
I was able to go to the no$ main website, download the windows version of no$gmb, and use it to debug when run in B/W mode - should be sufficient for you needs. F12 opens roms, F2 toggles break points, space traces, F3 steps over, Ctrl-G takes you to an address (or symbol), and Ctrl_B allows conditional break points (by far the most powerful feature for you to use.) For instance, (3000)! would set a read/write breakpoint on address 0x3000. (0300..03003)! sets on a range. As you are looking for specific address changes, this is what you want.
VBA-M has a bunch of debugging tools under "Tools" menu, including memory & tile inspectors and a disassembler. It even has support for GDB. I didn't test with any frontends like gdbgui, or VSCode's GDB support, so YMMV, but the other built-in tools look pretty decent.
Go to the releases section for a build for your platform (arch linux also has it in AUR, for easy install in package manager.)
Here it is running some of the tools on a Mac:
I just came back to C++ Builder after 5 or more years away. I seem to remember a nice tool where I could drag its pointer over the GUI of my running application and get lots of info about what was pointed at - handle, size, text, parent, children, etc
IIRC, if the exe include debug info I could also get the actual variable name as used in the source.
Does anyone know what program I am talking about? Thanks
There are several options:
Winspector
This is the program I've used the most of these Spy++ like programs, it has all the features I needed and worked without complaints on my developer machine, it has been quite some time since I've used it though. And the official site seems to be down for some reason ( http://www.windows-spy.com/ ) but luckily you can get a version from softpedia: http://www.softpedia.com/get/Security/Security-Related/Winspector.shtml
WinSpy
I have used this program for debugging, and finding out which messages was sent to different windows, it worked quite well, and was a good alternative to Spy++, besides it is free. You can download it from here: http://www.catch22.net/software/winspy
SpyStudio by Nektra
I have never tried this program but it actuall seems to do what you asks, and it is free. It can be found here: http://www.nektra.com/products/spystudio-api-monitor/
(source: nektra.com)
WinID
From what I've heard WinID should be able to perform much of the same, but I have never tried the program myself. You can download it from here: http://www.dennisbabkin.com/php/download.php?what=WinID this program is not compatible with Windows Vista though (and probably not Windows 7 either).
The challenging option
If there is a bit of a pioneer in you, you might want to create your own utility, although this is most likely not what you want, but if so you can check out these codeproject pages:
http://www.codeproject.com/KB/graphics/screen_capturing.aspx
http://www.codeproject.com/KB/dialog/windowfinder.aspx
These are all a lot like the Spy++ program that came with Visual Studio. Last but not least you might find other programs in this thread: I want Spy++ but I don't have Visual Studio
A very decent effort to revive the beloved but defunct winspector is window detective : http://windowdetective.sourceforge.net/
What you describe in the question sounds like TestComplete. http://www.automatedqa.com/products/testcomplete
If you compile the application as an "open application"(you add a few .pas from TestComplete to the project) then it will use the debug information and you will be able to see private data members of the classes and their value.
I've used minidumps on many game projects over the years and they seem to have about a 50% chance of having a valid call stack. What can I do to make them have better call stacks?
I've tried putting the latest dbghelp.dll in the exe directory. That seems to help some.
Is Visual Studio 2008 or 2010 any better? (I'm still on VS 2005).
The code I use looks like this sample.
One thing you can do to improve the accuracy of call stacks found in dumps is to use a debugger other than Visual Studio -- specifically, use WinDbg or another tool that uses the "Windows Debugger" debugging engine found in dbgeng.dll (as opposed to the "Visual Studio Debugger" debugging engine that Visual Studio uses).
In our experience, WinDbg is 100% reliable in producing good call stacks from the same dumps where Visual Studio produces unusable or wildly inaccurate call stacks. From what I can tell, in cases where an unhandled exception is the source of the crash WinDbg automatically performs the tricky process of reconstructing/recovering the exception callstack, but Visual Studio does not (or cannot?). The two debuggers use different heuristics for interpreting stacks
WinDbg can be daunting at first, so here's my quick guide on how to make it easier or even avoid having to use it directly.
A Mere Mortal's Guide To Extracting Good Callstacks
These are ordered from "fastest/easiest" to "slowest/most cryptic to interpret".
Easiest Option: use DbgDiag from Microsoft
This is a little-known tool that automates a lot of analysis of common problems, and it's simple enough to give to non-programmers or even customers. It's fast and nearly foolproof, and has become my "go to" tool for quickly analyzing an incoming crash dump.
Launch the "DebugDiag Analysis" application
Select the "CrashHangAnalysis" checkbox on the main page
Drag-and-drop your dump into the "Data files" pane on the main page
Click "Start Analysis"
After a few seconds to a few minutes it will spit out a nice .mhtml file containing an analysis of the problem, info about all the related thread, complete call stacks, etc. All hyperlinked and easy to use.
DebugDiag even automates some of the more complicated analysis that is possible but painful in WinDbg (like tracking down which of the 350 threads in your application is responsible for a deadlock).
Note: Chrome will not download or open .mhtml files for security reasons, so you must open in Internet Explorer or Microsoft Edge for it to be usable. This is annoying, and I've filed a request with the DebugDiag team (dbgdiag#microsoft.com) to change the format to plain HTML
Middle option: Install WinDbg as an alternate debugging engine for Visual Studio
Install Visual Studio if it's not yet installed. This needs to be done before the next step.
Install the Windows Driver Kit (WDK)
Launch Visual Studio, and (this part is important!) use the new "File -> Open -> Crash Dump..." option to open the dump. This will debug the crash dump using the Windows Debugger (if you instead drag-and-drop the dump on Visual Studio or use the standard "File -> Open -> File..." option to open the dump, it will debug it using the old Visual Studio debugging engine... so be careful to use the right option).
You should now be able to see the correct call stack and navigate around using the Visual Studio GUI, although some things work differently (the watch windows require using the unfamiliar WinDbg syntax, thread IDs are different, etc). Note: the Visual Studio UI may be very sluggish, especially if many threads are involved and the 'threads' or 'parallel stacks' windows are open.
Hardcore option: Use WinDbg directly
Launch WinDbg.exe
Drag-and-drop your dump into the WinDbg window
Type !analyze -v and press Enter. After a little bit of time WinDbg will spit out a crash call stack, and also its estimation of what the source of the problem is. If you're analyzing a deadlock, you can try !analyze -v -hang and WinDbg will often show you the dependency chain involved.
At this point you may have all the info you need! However, if you then want to examine the process state in the Visual Studio debugger you can take the following additional steps:
Open the crash dump in Visual Studio
Right-click in the callstack window and choose "Go to Disassembly"
Paste the hex address from the top line of WinDbg's output callstack into the "Address" bar of the Disassembly window and press enter. You're now at the location of the crash, looking at the disassembled code.
Right-click in the disassembly window and choose "Go To Source Code" to go to the source code for the location. Now you're looking at the source code at the crash site.
Note: all of the above require having correct symbol server paths configured, otherwise you won't be able to resolve the symbols in the call stacks. I recommend setting the _NT_SYMBOL_PATH environment variable so that it's automatically available to Visual Studio, WinDbg, and DebugDiag.
What's missing from your callstack? Do you have a bunch of addresses that don't resolve to valid function names (ie, 0x8732ae00 instead of CFoo:Bar())? If so, then what you need is to put your .PDBs where your debugger can find them, or set up a symbol server and set the "Symbol Paths" in the right-click context menu of the Modules pane.
We store every .PDB from every binary every time someone checks in a new Perforce changelist, so that when a dump comes back from anyone inside the office or any customer at retail, we have the .PDB corresponding to the version of the game they were running. With the symbol server and paths set, all I have to do is just double-click the .mdmp and it works every time.
Or do you have a call stack that appears to only have one function in it? Like, 0x8538cf00 without anything else above it in the stack? If so, then your crash is actually the stack itself being corrupted. If the return addresses in the backchain have been overwritten, naturally the debugger will be unable to resolve them.
Sometimes also you'll find that the thread that actually emits the minidump is not the one that threw the exception that caused the crash. Look in the Threads window to see if one of the other threads has the offending code in it.
If you are debugging a "Release" build -- that is to say, one compiled with all optimization flags turned on -- you will have to live with the fact that the debugger will have trouble finding local variables and some other data. This is because turning on optimizations means allowing the compiler to keep data on registers, collapse calculations, and generally do a variety of things that prevents data from ever actually being written to the stack. If this is your problem then you'll need to open up the disassembly window and chase the data by hand, or rebuild a debug binary and reproduce the problem where you can look at it.
Turn off Frame Pointer Optimization, if you need stack dumps. Frame pointers are used to explicitly define stack frames. Without them, the debugger has to deduce the location of each frame.
The code to record the minidump is unlikely to be relevant. The main things that a minidump records are module information (for getting symbols) and the full content of all thread stacks. Beyond that basic information (which is always recorded) nothing else matters.
Getting good symbols (including PE files) is crucial for stack walking. More details can be found here: https://randomascii.wordpress.com/2013/03/09/symbols-the-microsoft-way/
I find that Visual Studio is usually reliable at displaying call stacks. It automatically displays the relevant call stack from the exception record, and it makes changing threads easy so that you can see the call stacks of all threads. It does sometimes try to 'hide' details that it thinks might confuse you - whether that is good or bad depends on your skill level.
Windbg defaults to showing the call stack of the code that recorded the crash dump rather than the crashing call stack. Windbg requires that you go ".ecxr" or "!analyze -v" in order to see the crash stack. I find this annoying. Windbg also requires more configuration in order to be useful.
The two debuggers do have different stack walking heuristics. These heuristics are needed, for instance, if you call or return to address zero since there is no unwind information for that address. For 'clean' crashes where the failing instruction is in normal code these heuristics are less important.
The stack walking has almost certainly improved in the last ten years. VS 2015 Community Edition is very capable and is free so you might as well try it.
If you use windbg then you can try some experiments:
!vc7fpo - toggles some of the windbg heuristics.
!stackdbg d, 7, f - turns on windbg stack walk
k1 - walks one level of the stack, spitting diagnostics as controlled by !stackdbg
dds esp - dumps the raw contents of the stack, doing a symbol lookup on each pointer
If you upgrade to VS 2015 and still have problems then it is likely that the stack walking failures are specific to the crashes you are seeing. If a buffer overrun tromps the stack before crashing then the call stack will be irrevocably damaged. Your question has too little information about what failures you are seeing to give a definitive diagnosis. I find the stack displays of both debuggers fairly reliable, but I also usually understand why they sometimes fail and when that happens I can still extract the information that I need.
I dont use minidumps, but rather dump teh stack by "hand" into a logfile
(see www.ddj.com/cpp/185300443 and
How to Log Stack Frames with Windows x64).
I encounter a similar behavior like you do: Sometimes there is a valid call stack, sometimes there is not. In a minor number of cases the stack might be really corrupted. In maybe 1/3 of all cases the installed Exception handler is not called at all! I guess that its somehow a problem of the windows structured exception handling.
I'm using WinDBG occasionally to analyze problems in production environment, where VS cannot be installed. There's no doubt it's an extremely powerful tool, but using it is a bit annoying. Even though the product is frequently updated, its GUI goes back to the Win95 days or so, and its usability is accordingly. Having to fight the GUI to layout the windows the way I want, and having to remember all those textual commands, is just quite low a standard for a modern desktop application.
AFAIK, WinDBG is pretty much built on top of CDB, which is a command line debugger. Being so, it shouldn't be that hard to built a modern days GUI wrapper that will replace the existing dinosaur. Has anyone ever done that? Am I the only one having those mixed feelings toward WinDBG?
(BTW, I know I can create a dump and take it back to where I have VS, but I sometimes have to debug 64 bit processes, and I don't have a 64 bit dev machine. Sad, but true)
Consider the new WinDbg. (It's still in Preview). It also supports Time Travel Debugging.
You can install it from the Microsoft Store, or use the links here.
Here is what's new with Windbg Preview.
Have look at this if you fancy trying out a GUI to replace WinDbg.
EDIT:
Since SOS Assist is no longer available, this answer should be deleted. As this answer has been accepted, I personally cannot delete it. In that, please ignore my answer.
I guess thats too much to expect. With such a large number of commands that it has, it will not be trivial to have UI that displays everything in fancy controls. It might also make it bulkier, slower.
However it does provide you with controls that any user mode application debugger should have. It displays most frequently needed information like call stack, local variables, threads and so on in seperate windows.
But if you need more advanced debugging feature, you alwalys have the command interface.
WinDBG is pretty much it, no one has ever bothered to write their own UI for it. Even with its quirks I'm a fan because it's mostly command line driven. So, to each their own :)
Note that the VS 2011 Dev Preview basically integrates WinDBG support, so in the future VS will be the new WinDBG UI.
-scott