I open a particular crash dump in windbg, set the symbol path, load the exception record, and produce a stack walk with k. I get a wrong call stack, even though i know the symbols do match.
The output of lmi is full of lines like 00007ff7'6f3c0000 00007ff7'743f0000 WhateverModule T (private pdb symbols) WhateverModule.dll, note the T there. According to the documentation, it means:
The timestamp is missing, not accessible, or equal to zero.
This checks out with what is described in this answer, indeed i did not specify a binary path. Even though the dump seems to contain all the timestamps for all the relevant modules (for instance, lmDvmWhateverModule outputs
[...]
Timestamp: Mon Jun 15 13:18:45 2015 (557EB495)
CheckSum: 04F8AF3B
[...]
) involved in the stacktrace, the debugger still complains. If i feed it a correct binary search path, then i get a correct callstack.
My questions are:
Even though the timestamps are right there in the dump, windbg complains about them missing. Why?
I thought PDBs are matched with binaries comparing a GUID and a so-called age. Why does windbg refuse to load symbols blaming missing information that it actually has, and does not need? :)
My windbg version is 10.0.10240.9
I was under the assumption that unwind information is present in PDB files. I was wrong, it isn't (by default). However, it turns out there is a linker switch which instructs the linker to copy unwind data into PDBs: /DEBUGTYPE:PDATA
All you need to do is add /DEBUGTYPE:CV,PDATA to your linker switches (according to the documentation, /DEBUGTYPE:CV is the default for user mode programs compiled with /DEBUG).
After this, windbg has no problem doing stackwalks, even with binaries missing (made sure with !sym noisy and lmvm that binaries are indeed not found). Unfortunately, this doesn't work in Visual Studio (tried with 2015 Update 3).
Related
So, I primarily use VS IDE for debugging. I got a dump file and tried to do a postmortem on it. All of the DLLs loaded their respective PDBs except one and I don't know exactly why. This information would be helpful in determining if the dump file got corrupted in some way or if the client has a corrupted DLL.
I have also tried to use WinDbg to debug this, which I have some but not a lot of experience with. I updated the symbol paths to the directory that has the PDBs of the proper build and some others that it also might match up with as well. I loaded up the dump file and that same DLL is not having a matching PDB file found.
So the question is, what prevents a particular PDB not match with a dump file and how can I find out what that information is?
Symbols have a hash and a timestamp. Both need to match in order to load the symbols. In WinDbg, there's an option to force loading symbols that don't match (.symopt+ 0x40). Visual Studio doesn't have such an option, so you need to use chkmatch to make symbols match. Note that this is a dangerous operation, because it modifies the PDB file. You should create a backup copy and delete the modified file after you're done.
If you can't figure out what executable exactly is in the dump file, try .writemem <FileName> <Range> with the starting address of the executable and its size. See also How to retrieve assembly from a raw memory dump?.
For checking a dump file for corruption, I only know about DumpChk, which comes with WinDbg. AFAIK, the file format does not allow detection of single byte corruptions or similar.
I updated the symbol paths to the directory that has the PDBs
You should set up a symbol server. With a symbol server, there's no need to look for symbols or configure directories.
Crash Dump Debugging
Suspension of Disbelief: I know your first instinct may be to announce that "It isn't possible!" since this isn't how MS suggests solving the problem and they don't make tools available to accomplish it (easily). Forget that hurdle for a moment and ask yourself, "Is there some way to do this?"
I have a Windows binary I support in production that was built with debug information (MSVC 14.1 /Zi) but the debugger reports "was not built with debug information." This not wholly true—I think it was stripped by some tool during a post-build/pre-release process that I do not control, because I do have a PDB file for it. I have a memory dump from a run (crash) of that executable. Visual Studio and WinDbg both claim that the module has "no debug info," though WinDbg will attempt to load symbols for it anyway, if it finds some that match.
DBGHELP: No debug info for MyDll.dll. Searching for dbg file
DBGHELP: r:\crashes\symbols\MyDll.dbg - file not found
DBGHELP: r:\crashes\symbols\dll\MyDll.dbg - path not found
DBGHELP: r:\crashes\symbols\symbols\dll\MyDll.dbg - path not found
DBGHELP: MyDll.dll missing debug info. Searching for pdb anyway
*** WARNING: Unable to verify checksum for MyDll.dll
DBGENG: MyDll.dll has mismatched symbols - type ".hh dbgerr003" for details
DBGHELP: MyDll - private symbols & lines
r:\crashes\symbols\MyDll.pdb - unmatched
Here's the result of !chksym MyDll in WinDbg
MyDll.dll
Timestamp: 5CF5ABF8
SizeOfImage: 2437000
pdb sig: 0
age: 0
Loaded pdb is r:\crashes\symbols\MyDll.pdb
MyDll.pdb
pdb sig: 944F882B-73AE-45D0-9043-44C899BE09C5
age: 1
sig MISMATCH: MyDll.pdb and MyDll.dll
Is there a way to modify the memory dump (or the binary—assuming I can reproduce the problem) so as to re-inject whatever is necessary for the debugger to acknowledge that the symbols file I have matches the binary?
If your answer is "Nope!" I would appreciate some references to support that conclusion.
In the end, I want to be able to make sense of the call stack. Looking at variables/memory would be cool, too, but the call stack itself would be a gold mine in terms of debugging this problem.
Is "debug info" in this regard just a PE header (i.e. revealed by dumpbin /HEADERS) or is it more? Can I inject this header back into the binary/module? Is there more information that the debugger needs besides the symbols file to make sense of function addresses?
The tools for resolving symbol mismatches seem to work by modifying the PDB file. In my case, that doesn't seem to work because the binary module has no PDB signature in it to match to.
References
Crash dump - WinDbg - force PDB files to match doesn't work? (StackOverflow)
CHKMATCH (debuginfo.com)
How to Inspect the Content of a Program Database (PDB) File (codeproject.com)
microsoft-pdb (microsoft.com)
Corollary Questions
This is a problem I would like to solve one way or another (debugging a crash dump from a run of this "stripped" executable). So, it's possible that I'm asking the wrong question. Is there another way to solve this problem?
Can I decode memory addresses "outside" the Visual Studio debugger using the data in the PDB file?
Is there some other way to have the debugger use the symbols in a PDB file as if there was a match (assuming the module "has no debug info")?
If you can patch the PDB signature GUID and the age field from the PDB back into the binary, that might be enough to convince the tools that you have the right symbols. It seems likely those fields (or the section they live in) were cleared by the stripping tool.
The LLVM project has some documentation on the PDB format. Specifically, this section has some brief mentions about how PDBs and binaries are matched by the debuggers. And this section may tell you how to find (or recreate) the PDB Guid and age fields in the binary.
There are also some flags in the PE header (like IMAGE_FILE_LINE_NUMS_STRIPPED) that signal the debugger that information has been stripped. You might have to reset those to convince the debugger to even try to look for the PDB.
I don't know what else the stripping tool may have removed. In 32-bit binaries, I believe the frame-pointer opimization (FPO) data that's required for reliable stack unwinding is included in both the executable and the PDB. If the FPO data was stripped from the executable, I'd expect the debugger to use the data from the PDB, but I've not confirmed that. If I had to guess, I'd expect Windbg is more likely to rely on the PDB than the VS debugger.
In x86_64 binaries, basic stack unwinding is simpler (e.g., you should almost always be able to get a stack trace without much unwinding data). But if you want to examine locals and arguments, you do need some unwinding data, and that's stored in the PDB.
Sorry I have only clues rather than answers. I hope this is helpful.
Update 1: From your ChkMatch link, I read:
ChkMatch is capable of making an executable and PDB file match if they have different signatures but the same age (see this article for more information about PDB signature and age). If the age differs, the tool cannot make the files match. Signature and age can be displayed using -c option.
In the display you quoted, the ages are different (0 in executable and 1 in PDB). This seems the likely cause of why chkmatch -m won't solve your problem.
I'm debugging a problem at present in Windbg from a dump. I've got the correct pdbs and I can view the locals etc quite happily. However, the source code I have (which I've pulled from the VCS branch from which this release supposedly came from) appears to be off by several lines in some parts of the stack I'm looking at. I've seen instances where it's off by 1 before, but not 3/4 lines.
What causes this? Is there any definitive way I can check that I've got the right source files?
Are you looking at debug or non optimised version of release code?
Code optimisation may cause the line shift you are seeing so you should recompile with /Od C++ optimization set to 'disabled' and see if this corrects what you see.
WinDbg uses the same method as Visual Studio to check if the source file you are viewing/setting breakpoints with matches the pdbs so it should warn you (I think it does this I cannot verify).
Besides you can verify the pdbs if not the source files using:
!itoldyouso myDLL
additionally you can open the source file in another window, during stepping and it should put a magenta line at the line it thinks the current call is at, this should be correct and behave the same as visual studio.
I have received a crash dump file from a customer, and I am attempting to debug the dump file using VS2010. I have retrieved the source code we used for the build along with the generated symbol files, and I've provided VS with the path to the directory that contains them; everything seems to be in place. When I attempt to debug, however, VS reports that there is "no matching binary" for the .exe or any of the .dll's contained in that directory. I've used chkmatch to compare a few of the binaries and symbol files, and it reports that they match, time stamp and signature and all. I've searched around online and no one seems to have had as much trouble with this as I am encountering, and all the solutions I've found are things that I've already done.
Hopefully I'm overlooking something simple.
What am I missing? How can I ensure that VS matches the binaries to the symbol files?
You can use WinDbg
Download the tool (http://msdn.microsoft.com/en-us/windows/hardware/gg463016, you can use version 6.11.1.404 in order to avoid downloading the whole SDK) and install it.
start WinDbg and load the dump file (File > Open crash Dump).
specify the location of the symbols (File > Symbol File Path).
activate traces on symbols loading (!sym noisy at the command prompt)
unload all symbols (.reload /u at the command prompt).
load symbols for a module you have problem with (ld at the command prompt).
the engine will tell you where it loads symbols and why it doesn't match.
you can also have detailed informations on the module contained in the dump file (lm vm ), cinluding the timestamp of the module.
Hope this helps.
Another thing to check: I had problems if binaries and symbols for other build versions of the same product were in the symbol path.
VS (I'm using 2015) doesn't seem to search all the paths and choose the right files - I had to add symbol paths specific to the crash dump I'm debugging and move these "up" in the symbol path list for them to be found first and used.
When using WinDbg, where should the private symbol files (pdb?) be placed?
My situation is: I have a DLL which I want to debug. I have the source code and symbol files for this DLL. This DLL is called by another DLL (which I don't have symbols or source for) which, in turn, is called by an EXE (which I also don't have symbols or source for).
My problem is that I am getting a warning that says
*** WARNING: Unable to verify checksum for C:\TheProgram\SomeSubfolder\AnotherSubfolder\MyDll.dll
This warning I think is the reason why I am getting the following type of messages in the call stack:
MyDll!AClass::AFunction+SomeHexAddress
My file structure looks something like this:
The exe: C:\TheProgram\program.exe
The calling dll: C\TheProgram\SomeSubfolder\caller.???
My DLL that I want to debug: C:\TheProgram\SomeSubfolder\AnotherSubfolder\MyDll.dll
Note: I set Symbol File path and the Source file path to where the debug DLL was generated, in my workspace on a different drive from the exe.. But I did copy the pdb + map files and put it on the dll that I wanted to debug..
Sorry for the late reply.
In your post you mention that you are seeing the following error message.
*** WARNING: Unable to verify checksum for C:\TheProgram\SomeSubfolder\AnotherSubfolder\MyDll.dll
You also ask the question, "where do I put my symbols for my DLL in the symbol path?"
Here is a response for the first problem:
Steps to identify mismatched symbols.
!sym noisy
.reload
x MyDll!*class*
*This reloads your dll, alternatively you can type kb to display the call stack of the DLL which should load it as well.
!sym quiet
*Reset's back to original quiet symbol loading
Also you can run
0:001> lmv m myDll *(and examine the Checksum)
Note: If you have a checksum, then Windbg can match the checksum of the DLL against the checksum of the PDB. Every development environment has a different way to generate a checksum.
Here is the response for the questions about where to put the PDBs
If you have MyDll.pdb added to a symbol store then you can use the following syntax
.sympath SRV*c:\symcache*http://msdl.microsoft.com/download/symbols
As Roger has suggested above...
However if you just have the PDB locally, you may want to put the path to the PDB first before going out to the symbol server like this
.sympath C:\TheProgram\SomeSubfolder\AnotherSubfolder\;SRV*c:\symcache*http://msdl.microsoft.com/download/symbols
This way Windbg should look local to your SomSubFolder dir before trying to use the Symbols Server cache.
Thanks,
Aaron
It does not matter where you put private symbol files as long as you're able to tell the debugger where they are.
The warning you're seeing does not have any effect on the stack trace, but the fact you're missing symbols for caller.DLL and app.EXE does.
Configuring symbols in windbg (locally) is as simple as using:
.sympath[+] path_to_pdbs
*and
.symfix+ path_to_system_pdb_store
You seeing:
MyDll!AClass::AFunction+SomeHexAddress
actually means nothing as long as SomeHexAddress is reasonable (and provided that MyDll.pdb has been found and loaded!) - it looks like a proper call stack entry.
Now, my question would be, what is the problem that you're stuck with?
P.S. you don't need .map file with windbg.
As part of our build process, we copy the private PDB files and the released EXE/DLL files to a symbol server. At its simplest, this is just a UNC path, but you can configure it for access using HTTP.
To copy your output files, use the SYMSTORE.EXE program.
Then, configure your debugger (we use Visual Studio and WinDbg) to look in that path. For WinDbg, the simplest way to do this is to set an environment variable:
_NT_SYMBOL_PATH=
SRV*C:\WebSymbols*http://msdl.microsoft.com/download/symbols;
\\symsvr\Symbols
(that should all be on one line)
This configures WinDbg to look on the Microsoft Symbol Server (caching the files in C:\WebSymbols) and also to look in a local symbol store (\\symsvr\Symbols).
We also use the Source Server tools to store SVN details in the PDB file, meaning that we can get back to the exact source file used to build a particular release. Look in ...\Debugging Tools for Windows (x86)\srcsrv.
One option is to leave the symbol files where they are (i.e. in the build output folder) and then use -y WinDbg command line option to locate these files. Using this approach should guarantee that the symbol files are always be up to date.
From the Microsoft Help:
-y SymbolPath
Specifies the symbol search path. Separate multiple paths with a
semicolon (;). If the path contains spaces, it should be enclosed
in quotation marks. For details, and for other ways to change this
path, see Symbol Path.
As it turned out, my target machine - provisioned inside Visual Studio - did not get the latest build upon deploying the to it, hence a "driver.sys has mismatched symbols" error.
Basically deploying did not replace the driver with the modified version of it for me. Use devcon tool to properly install it and Windbg will be happy again.