With the dump debugging support in .NET 4.0 we are looking into automatically (after asking the user of course :) creating minidumps of C# program crashes to upload them to our issue tracking system (so that the minidumps can assist in resolving the cause of the crash).
Everything is working fine when using the WithFullMemory minidump type. We can see both stack and heap variables. Unfortunately the (zipped) dumps are quite large even for small C# programs.
If we use the "Normal" minidump type we get a very small dump, but not even stack variable information is available in the managed debugger. In fact, anything less than WithFullMemory seems quite useless in the managed debugger. We have made a few attempts at using a MINIDUMP_CALLBACK_ROUTINE to limit the included module information to our own modules, but it seems that it has almost no effect on a managed dump but still manages to break the managed debugging?
Does anyone have any tips on how to trim the minidump while keeping it useful for managed debugging?
I use the following flags to save space will generating useful minidumps for C++ applications:
MiniDumpWithPrivateReadWriteMemory |
MiniDumpWithDataSegs |
MiniDumpWithHandleData |
MiniDumpWithFullMemoryInfo |
MiniDumpWithThreadInfo |
MiniDumpWithUnloadedModules
The flag values are specified in DbgHelp.h and would need to be marshaled into C#. The dump is further restricted by specifying a CallbackRoutine.
Just fyi, as mentioned above ClrDump looks very cool but it appears it only works with the 1.1. and 2.0 runtimes.
With all due respect, I STRONGLY encourage you to sign-up for a Microsoft WinQual account, register your applications with Microsoft.
http://www.microsoft.com/whdc/winlogo/maintain/StartWER.mspx
This will allow you to not only take advantage of Microsoft's extensive crash collection and analysis services (for free!), but will also allow you to publish fixes and patches for your applications through Windows' built-in error reporting facilties.
Further, by participating in the WinQual program, enterprises who deploy your app and who employ an in-house Windows Error Reporting system will be able to collect, report and receive patches for your app too.
Another benefit is that employing WinQual, you're one step closer to getting your app logo certified!
Every OEM & ISV I've worked with who uses WinQual saves an ENORMOUS amount of effort and expense compared to rolling their own crash collection and reporting system.
I wrote an email to author of ClrDump asking a question what MINIDUMP_TYPE parameters his tool used to create dumps in 'min' mode. I posted his answer here: What is minimum MINIDUMP_TYPE set to dump native C++ process that hosts .net component to be able to use !clrstack in windbg
ClrDump might help you out.
ClrDump is a set of tools that allow to produce small minidumps of
managed applications. In the past, it was necessary to use full dumps
(very large in size) if you needed to perform post-mortem analysis of
a .NET application. ClrDump can produce small minidumps that contain
enough information to recover the call stacks of all threads in the
application.
Related
Is there any way to figure out which application is using up all the desktop heap memory?
For an explanation of 'desktop heap' see this MSDN blog.
EDIT: If you don't know what "desktop heap memory" is please don't answer.
EDIT2: if you don't know what "desktop heap memory" is don't vote to close the question.
Disclaimer: I work for Microsoft, and these are my personal thoughts and experiences which may or may not reflect current or future Microsoft policies/procedures/etc.
I know this is old, but since I stumbled across it I wanted to provide some feedback/answers.
Desktop Heap Monitor does not work on Vista+. DHeapMon depended on the ability to patch kernel binaries to track desktop heap- this behavior is blocked, which is why it doesn't work.
To monitor desktop heap yourself, all you can do is look at user objects (unfortunately).
To get a complete answer, you can open a support ticket with Microsoft and with a kernel or complete dump (NOT a process dump!) and they can inspect the heap and provide you some information. It is extracted from the dump file via a debugger extension, which I wrote.
Limitations:
Only works from kernel mode. Even though a desktop heap mapping exists in user mode, the extension does not support it. Even if I added support for user mode debugging of desktop heap, it would be limited as some of the related data is only in kernel mode. Changes on recent builds of Windows 10 might restrict it further, so I have not invested time to add user mode support.
It can't measure what isn't in the dump. In other words, if the desktop heap is missing pages it will do its best but it will obviously lose accuracy.
Does not support Windows 10 build 1809 or Server 2019, yet, due to changes to the heap manager.
Happy coding!
According to this blog post you should look for the user objects.
To find the bad application you can configure the USER Objects column within Process Explorer. Then sort descending by user objects. User Objects are all GUI objects like Windows, Toolbars, Icons, Cursors ... The process with the highest values is very likely the bad one.
Direct link to download: DHeapMon from microsoft.
If you want to do it within your program, C++ can use WMI to find the culprit.
I have a .net application which I don't have source too. I'm interested in determining what files this application is accessing. I thought that a debugger would be a good way to do this. I work only in the linux world and know nothing about windows debuggers. I feel like there should be a way to do this with windbg, I know how I would go about it with gdb but not with windbg. Any help or advice is much appreciated.
If you just want to monitor file system activity for a given process then you won't find anything better than Process Monitor. It does lots more too and it's free from Sysinternals (now Microsoft).
Process Monitor is an advanced
monitoring tool for Windows that shows
real-time file system, Registry and
process/thread activity. It combines
the features of two legacy
Sysinternals utilities, Filemon and
Regmon, and adds an extensive list of
enhancements including rich and
non-destructive filtering,
comprehensive event properties such
session IDs and user names, reliable
process information, full thread
stacks with integrated symbol support
for each operation, simultaneous
logging to a file, and much more. Its
uniquely powerful features will make
Process Monitor a core utility in your
system troubleshooting and malware
hunting toolkit.
Stu is right if you just want to know what files are being accessed.
But if you want to know more about debugging .net apps, you can place break points on .net code when you have SOS.dll loaded into windbg. It's more involved than I can describe here, but you can Google windbg and SOS.DLL to find more info. Tess Ferrandez has a great blog that you may find useful, too: http://blogs.msdn.com/b/tess/
What rarely used debugging tools you found useful ?
My recent debugging situation on Visual Studio required trapping the breakpoint on fresh built 32-bit DLL, which was loaded by GUI-less executable, which was spawned by COM+ server on remote x64 machine, which was called through RPC from actual GUI. As usual, all worked well on all 32 bit machine, but kept failing on "machine other than development one". So remote debugging was inevitable.
So after scratching the head beaten against wall for 2 days, I have added 10 sec delay into DLL attach entry point and used Microsoft Remote Debugger wich I never used before. It saved my day.
Another favorite: Java JMX console as a performance "debugging" tool. You can see all threads, memory chart, have a snapshot of any thread stack any time you click. Clicking several times helps to find what exactly is slow in J2EE application.
Process Monitor and other Mark Russinovich's tools.
A logic analyzer plugged to CPU pins and able to disassemble executed code. I tracked a bug in the boot sequence of an embedded system.
I find printf to be the most useful.
These - in my experience at least - do not seem to be the intuitive first choice for many when debugging apps accessing a database (i.e. the majority), that perhaps they should be :
SQL Profiler (SQL Server)
TKPROF (Oracle)
Another interesting combination was using eclipse running in a virtual machine, accessing a remote server, attaching to the Tomcat process there; and doing it from two different machines to debug two different packages simultaneously.
All time favorite is depends.exe, for finding out why a dll or exe is not starting http://dependencywalker.com/
For performance, at my former job we used to have really simple to use C++ macro's that did statistics on runtime function calls. This is so much better than a profiler, because you can use it from your regular IDE, and it allows you to zoom in on the code you are optimizing.
In my new job, I wrote a C# version of the same idea.
WinDbg and other lower level debuggers are the ultimate weapon if you know the tricks and tips.
For Windows/.Net development I am always using Debugview and Ildasm.
I am facing a severe problem with my program, which gets reproduced only in the customer place. Putting logs, are not helping as I doubt the failure is happening in a third party dll. For some reasons, I couldn't get help from the library provider. I am thinking of producing a dump at the point of failure, so that to analyze it offline. Is this a recommended practice? Or any alternatives?
Yes, this is something that every program should have and utilize as often as possible.
I suggest that you don't use third party libraries. Create your own dumps instead. It's very simple and straight forward. You basically need to do the following:
Your program needs to access dbghelp.dll. It's a windows dll that allows you to create human readable call stacks etc. The debugger uses this dll to display data in your process. It also handles post mortem debugging, i.e. dumps of some sort. This dll can safely be distributed with your software. I suggest that you download and install Debugging Tools for Windows. This will give you access to all sorts of tools and the best tool WinDbg.exe and the latest dbghelp.dll is also in that distribution.
In dbghelp.dll you call e.g. MiniDumpWriteDump(), which will create the dump file and that's more or less it. You're done. As soon as you have that file in your hands, you can start using it. Either in the Visual Studio Debugger, which probably even might be associated with the .dmp file extension, or in WinDbg.
Now, there are a few things to think of while you're at it. When checking dump files like this, you need to generate .pdb files when you compile and link your executable. Otherwise there's no chance of mapping the dump data to human readable data, e.g. to get good callstacks and values of variables etc. This also means that you have to save these .pdb files. You need to be able to match them exactly against that very release. Since the dump files are date stamped with the date stamp of the executable, the debugger needs the exact pdb files. It doesn't matter if your code hasn't changed a single bit, if the .pdb files belong to another compilation session, you're toast.
I encourage every windows win32 developer to check out Oleg Starodumov's site DebugInfo.com. It contains a lot of samples and tutorials and how you can configure and tune your dump file generation. There are of course a myriad of ways to exclude certain data, create your custom debug message to attach to the dump etc.
Keep in mind that minidumps will contain very limited information about the application state at exception time. The trade off is a small file (around 50-100 kB depending on your settings). But if you want, you can create a full dump, which will contain the state of the whole application, i.e. globals and even kernel objects. These files can be HUGE and should only be used at extreme cases.
If there are legal aspects, just make sure your customers are aware of what you're doing. I bet you already have some contract where you aren't supposed to reveal business secrets or other legal aspects. If customers complain, convince them how important it is to find bugs and that this will improve the quality of the software drastically. More or less higher quality at the cost of nothing. If it doesn't cost them anything, that's also a good argument :)
Finally, here's another great site if you want to read up more on crash dump analysis: dumpanalysis.org
Hope this helps. Please comment if you want me to explain more.
Cheers !
Edit:
Just wanted to add that MiniDumpWriteDump() requires that you have a pointer to a MINIDUMP-EXCEPTION-INFORMATION (with underscores) struct. But the GetExceptionInformation() macro provides this for you at time of exception in your exception handler (structured exception handling or SEH):
__try {
}
__except (YourHandlerFunction(GetExceptionInformation())) {
}
YourHandlerFunction() will be the one taking care of generating the minidump (or some other function down the call chain). Also, if you have custom errors in your program, e.g. something happens that should not happen but technically is not an exception, you can use RaiseException() to create your own.
GetExceptionInformation() can only be used in this context and nowhere else during program execution.
Crash dumps are a pretty common troubleshooting method and can be very effective, especially for problems that only reproduce at the customer's site.
Just make sure the customer/client understands what you're doing and that you have permission. It's possible that a crash dump can have sensitive information that a customer may not want (or be permitted) to let walk out the door or over the wire.
Better than that there are libraries that will upload crash data back you.
BugDump and BugSplat
And there's the Microsoft way:
http://msdn.microsoft.com/en-us/library/aa936273.aspx
Disclaimer: I am not a lawyer, nor do I pretend to be one, this is not legal advice.
The data you can include in logs and crash dumps also depend on what domain you are working in. For example, medical equipment and patient information systems often contain sensitive data about patients that should not be visible to unauthorized persons.
The HIPAA Privacy Rule regulates
the use and disclosure of certain
information held by "covered entities"
(...) It establishes regulations for
the use and disclosure of Protected
Health Information (PHI). PHI is any
information held by a covered entity
which concerns health status,
provision of health care, or payment
for health care that can be linked to
an individual.[10] This is interpreted
rather broadly and includes any part
of an individual's medical record or
payment history. --Wikipedia
It should not be possible to link health information to an individual. The crash dumps and logs should be anonymized and stripped of any sensitive information, or not sent at all.
Maybe this does not apply to your specific case, so this is more of a general note. I think it applies to other domains that handle sensitive information, such as military and financial, and so on.
Basically the easiest way to produce a dumpfile is by using adplus. You don't need to change your code.
Adplus is part of the debugging tools for windows, as mentioned in the article above.
Adplus is basically a huge vbscript automation of windbg.
What you have to do to use adplus:
Download and install Debugging tools for windows to c:\debuggers
start your application
open a commandline and navigate to c:\debuggers
run this line "adplus -crash your_exe.exe"
reproduce the crash
you'll get a minidump with all the information you need.
you can open the crash dump in your favorite debugger.
within windbg, the command "analyze -v" helped me in at least 40% of all the crashes that only happened at customer site and were not reproducible in house.
I have a crash dump of an application that is supposedly leaking GDI. The app is running on XP and I have no problems loading it into WinDbg to look at it. Previously we have use the Gdikdx.dll extension to look at Gdi information but this extension is not supported on XP or Vista.
Does anyone have any pointers for finding GDI object usage in WinDbg.
Alternatively, I do have access to the failing program (and its stress testing suite) so I can reproduce on a running system if you know of any 'live' debugging tools for XP and Vista (or Windows 2000 though this is not our target).
I've spent the last week working on a GDI leak finder tool. We also perform regular stress testing and it never lasted longer than a day's worth w/o stopping due to user/gdi object handle overconsumption.
My attempts have been pretty successful as far as I can tell. Of course, I spent some time beforehand looking for an alternative and quicker solution. It is worth mentioning, I had some previous semi-lucky experience with the GDILeaks tool from msdn article mentioned above. Not to mention that i had to solve a few problems prior to putting it to work and this time it just didn't give me what and how i wanted it. The downside of their approach is the heavyweight debugger interface (it slows down the researched target by orders of magnitude which I found unacceptable). Another downside is that it did not work all the time - on some runs I simply could not get it to report/compute anything! Its complexity (judging by the amount of code) was another scare-away factor. I'm not a big fan of GUIs, as it is my belief that I'm more productive with no windows at all ;o). I also found it hard to make it find and use my symbols.
One more tool I used before setting on to write my own, was the leakbrowser.
Anyways, I finally settled on an iterative approach to achieve following goals:
minor performance penalties
implementation simplicity
non-invasiveness (used for multiple products)
relying on as much available as possible
I used detours (non-commercial use) for core functionality (it is an injectible DLL). Put Javascript to use for automatic code generation (15K script to gen 100K source code - no way I code this manually and no C preprocessor involved!) plus a windbg extension for data analysis and snapshot/diff support.
To tell the long story short - after I was finished, it was a matter of a few hours to collect information during another stress test and another hour to analyze and fix the leaks.
I'll be more than happy to share my findings.
P.S. some time did I spend on trying to improve on the previous work. My intention was minimizing false positives (I've seen just about too many of those while developing), so it will also check for allocation/release consistency as well as avoid taking into account allocations that are never leaked.
Edit: Find the tool here
There was a MSDN Magazine article from several years ago that talked about GDI leaks. This points to several different places with good information.
In WinDbg, you may also try the !poolused command for some information.
Finding resource leaks in from a crash dump (post-mortem) can be difficult -- if it was always the same place, using the same variable that leaks the memory, and you're lucky, you could see the last place that it will be leaked, etc. It would probably be much easier with a live program running under the debugger.
You can also try using Microsoft Detours, but the license doesn't always work out. It's also a bit more invasive and advanced.
I have created a Windbg script for that. Look at the answer of
Command to get GDI handle count from a crash dump
To track the allocation stack you could set a ba (Break on Access) breakpoint past the last allocated GDICell object to break just at the point when another GDI allocation happens. That could be a bit complex because the address changes but it could be enough to find pretty much any leak.