I have a mindmp file from a target's application crash. Is it possible for me to rebuild the dll/pdb files for a version of software and have windbg load symbols correctly?
My problem is that our pdb files are only kept for major releases (unfortunately). This is a daily build, which I can rebuild myself, but I'm getting tripped up on errors.
With !sym noisy on:
"image header does not match memory image header."
DBGENG: C:\...\XXX.dll image header does not match memory image header.
DBGENG: XXX.dll - Partial symbol image load missing image info
DBGHELP: Module is not fully loaded into memory.
DBGHELP: Searching for symbols using debugger-provided data.
DBGHELP: C:\...\XXX.pdb - mismatched pdb
Note I've build the pdb with the dll, they are from the same RELEASE directory (should I be building debug?)
Theses are release builds (as release builds are installed on the target and crashing) should I be somehow using the debug build dlls to get more symbol information?
The ChkMatch utility is designed for this exact scenario.
As long as you have the original .EXE, you can recompile the sources (with the same compiler and compiler settings) and patch the new .PDB to match the old .EXE.
In this example, OriginalExecutable.exe is the executable that no longer has a .PDB file, and RebuiltPDB.pdb is one that has been produced by rebuilding the original source.
chkmatch -m OriginalExecutable.exe RebuiltPDB.pdb
Now, as long as the two files have their original names, The debugger should accept them as a matching pair.
In my experience probably not.
If you have the exact build directory and build with the exact same compiler settings then this might work. You definitely will not be able to load symbols from a debug build against a release crash dump.
You will need to turn on the 'load anything' options: .symopt+0x40 to get windbg to ignore the timestamp differences.
if you still have the exact source code the image was compiled from, then rebuild it producing a new pdb file and then instruct WinDbg to forcibly load this pdb when you open the crash dump - it worked once in my practice.
PDB files are tied to their EXE files by a GUID and an "age" (it's a sequence number). These are embedded in the EXE, and into the PDB. The GUID is regenerated on each complete build, and the "age" is changed on each incremental build.
The debugger uses these to ensure that it's looking at the correct PDB for the EXE file.
I didn't know about the "chkmatch" tool mentioned by SteveMan, but I suspect that it works by patching up the GUID/age so that they match.
This is too late to help Doug, but for the sake of anyone who comes across this question, another thread (Is it possible to load mismatched symbols in Visual Studio?) pointed out a way to get WinDbg to accept mismatched .PDB files
.symopt_0x40
Related
So, I primarily use VS IDE for debugging. I got a dump file and tried to do a postmortem on it. All of the DLLs loaded their respective PDBs except one and I don't know exactly why. This information would be helpful in determining if the dump file got corrupted in some way or if the client has a corrupted DLL.
I have also tried to use WinDbg to debug this, which I have some but not a lot of experience with. I updated the symbol paths to the directory that has the PDBs of the proper build and some others that it also might match up with as well. I loaded up the dump file and that same DLL is not having a matching PDB file found.
So the question is, what prevents a particular PDB not match with a dump file and how can I find out what that information is?
Symbols have a hash and a timestamp. Both need to match in order to load the symbols. In WinDbg, there's an option to force loading symbols that don't match (.symopt+ 0x40). Visual Studio doesn't have such an option, so you need to use chkmatch to make symbols match. Note that this is a dangerous operation, because it modifies the PDB file. You should create a backup copy and delete the modified file after you're done.
If you can't figure out what executable exactly is in the dump file, try .writemem <FileName> <Range> with the starting address of the executable and its size. See also How to retrieve assembly from a raw memory dump?.
For checking a dump file for corruption, I only know about DumpChk, which comes with WinDbg. AFAIK, the file format does not allow detection of single byte corruptions or similar.
I updated the symbol paths to the directory that has the PDBs
You should set up a symbol server. With a symbol server, there's no need to look for symbols or configure directories.
Crash Dump Debugging
Suspension of Disbelief: I know your first instinct may be to announce that "It isn't possible!" since this isn't how MS suggests solving the problem and they don't make tools available to accomplish it (easily). Forget that hurdle for a moment and ask yourself, "Is there some way to do this?"
I have a Windows binary I support in production that was built with debug information (MSVC 14.1 /Zi) but the debugger reports "was not built with debug information." This not wholly true—I think it was stripped by some tool during a post-build/pre-release process that I do not control, because I do have a PDB file for it. I have a memory dump from a run (crash) of that executable. Visual Studio and WinDbg both claim that the module has "no debug info," though WinDbg will attempt to load symbols for it anyway, if it finds some that match.
DBGHELP: No debug info for MyDll.dll. Searching for dbg file
DBGHELP: r:\crashes\symbols\MyDll.dbg - file not found
DBGHELP: r:\crashes\symbols\dll\MyDll.dbg - path not found
DBGHELP: r:\crashes\symbols\symbols\dll\MyDll.dbg - path not found
DBGHELP: MyDll.dll missing debug info. Searching for pdb anyway
*** WARNING: Unable to verify checksum for MyDll.dll
DBGENG: MyDll.dll has mismatched symbols - type ".hh dbgerr003" for details
DBGHELP: MyDll - private symbols & lines
r:\crashes\symbols\MyDll.pdb - unmatched
Here's the result of !chksym MyDll in WinDbg
MyDll.dll
Timestamp: 5CF5ABF8
SizeOfImage: 2437000
pdb sig: 0
age: 0
Loaded pdb is r:\crashes\symbols\MyDll.pdb
MyDll.pdb
pdb sig: 944F882B-73AE-45D0-9043-44C899BE09C5
age: 1
sig MISMATCH: MyDll.pdb and MyDll.dll
Is there a way to modify the memory dump (or the binary—assuming I can reproduce the problem) so as to re-inject whatever is necessary for the debugger to acknowledge that the symbols file I have matches the binary?
If your answer is "Nope!" I would appreciate some references to support that conclusion.
In the end, I want to be able to make sense of the call stack. Looking at variables/memory would be cool, too, but the call stack itself would be a gold mine in terms of debugging this problem.
Is "debug info" in this regard just a PE header (i.e. revealed by dumpbin /HEADERS) or is it more? Can I inject this header back into the binary/module? Is there more information that the debugger needs besides the symbols file to make sense of function addresses?
The tools for resolving symbol mismatches seem to work by modifying the PDB file. In my case, that doesn't seem to work because the binary module has no PDB signature in it to match to.
References
Crash dump - WinDbg - force PDB files to match doesn't work? (StackOverflow)
CHKMATCH (debuginfo.com)
How to Inspect the Content of a Program Database (PDB) File (codeproject.com)
microsoft-pdb (microsoft.com)
Corollary Questions
This is a problem I would like to solve one way or another (debugging a crash dump from a run of this "stripped" executable). So, it's possible that I'm asking the wrong question. Is there another way to solve this problem?
Can I decode memory addresses "outside" the Visual Studio debugger using the data in the PDB file?
Is there some other way to have the debugger use the symbols in a PDB file as if there was a match (assuming the module "has no debug info")?
If you can patch the PDB signature GUID and the age field from the PDB back into the binary, that might be enough to convince the tools that you have the right symbols. It seems likely those fields (or the section they live in) were cleared by the stripping tool.
The LLVM project has some documentation on the PDB format. Specifically, this section has some brief mentions about how PDBs and binaries are matched by the debuggers. And this section may tell you how to find (or recreate) the PDB Guid and age fields in the binary.
There are also some flags in the PE header (like IMAGE_FILE_LINE_NUMS_STRIPPED) that signal the debugger that information has been stripped. You might have to reset those to convince the debugger to even try to look for the PDB.
I don't know what else the stripping tool may have removed. In 32-bit binaries, I believe the frame-pointer opimization (FPO) data that's required for reliable stack unwinding is included in both the executable and the PDB. If the FPO data was stripped from the executable, I'd expect the debugger to use the data from the PDB, but I've not confirmed that. If I had to guess, I'd expect Windbg is more likely to rely on the PDB than the VS debugger.
In x86_64 binaries, basic stack unwinding is simpler (e.g., you should almost always be able to get a stack trace without much unwinding data). But if you want to examine locals and arguments, you do need some unwinding data, and that's stored in the PDB.
Sorry I have only clues rather than answers. I hope this is helpful.
Update 1: From your ChkMatch link, I read:
ChkMatch is capable of making an executable and PDB file match if they have different signatures but the same age (see this article for more information about PDB signature and age). If the age differs, the tool cannot make the files match. Signature and age can be displayed using -c option.
In the display you quoted, the ages are different (0 in executable and 1 in PDB). This seems the likely cause of why chkmatch -m won't solve your problem.
When building a large C++/Fortran app, I recently started getting LNK4099 errors for virtually every C++ object file. For example
Cfile.obj : warning LNK4099: PDB 'lnk{3FE844DB-7378-4485-9D93-6B1B48386536}.tmp' was not found with 'Cfile.obj' or at 'C:MyApp\x64\Debug\lnk{3FE844DB-7378-4485-9D93-6B1B48386536}.tmp'; linking object as if no debug info
Unlike a number of previous posts, this is not due to a library missing PDB info; the files with the errors are all my own source code, and freshly built.
This is Visual Studio 2015, building under 64-bit Windows 7. The problem occurs for both debug and release builds. The options for the debug build are C++: /Zi / Od; Linker: /DEBUG, Generate Full Program Database File.
The app is C++, using a Fortran library, created by Intel Fortran XE2017, and built with /debug:full. Linking to the Microsoft libraries (MFC, msimg32.lib, nafxcwd.lib, libcmtd.lib, etc) is static.
If I compile a single C++ source file AFile.cpp using F7 (Build/Compile) and then build the project, I get all the same errors EXCEPT there is none for AFile.obj. The debugging information for AFile is in the app's PDB (I can set breakpoints). Files that have the errors are missing debug info in the app's PDB (as the error message says), and I cannot set breakpoints.
What setting or configuration could be causing this mysterious behavior? I do not have this issue with smaller projects.
The LNK4099 documentation shows a DUMPBIN command that can be used to list the full path name of a .pdb file associated with an object file...
dumpbin /section:.debug$T /rawdata objectname.obj
It might be interesting to examine AFile.obj from your F7 experiment and another .obj file generated by the project build to see how the PDB file names differ (if they differ).
Assuming you're doing a build, not rebuild, of the project in your F7 experiment it might also be interesting to see what happens if you compile AFile.cpp with F7 and then do a full rebuild of the project. A full rebuild would recompile AFile.cpp where a regular build wouldn't.
Once you know what you should be looking for you can start trying to figure out if your build is: not creating the PDB files; creating them in the wrong place; creating them with the wrong name; or removing them after they're created.
UPDATE
I should have added the caveat that /PDBALTPATH can set the string shown by the DUMPBIN command above to a value different from the actual PDB file's pathname.
Per #Frank Boyne's suggestion, I found that
dumpbin /section:.debug$T /rawdata *.obj
lists the same result for all the object files, namely 0x1B byes of binary data, followed by the full path to App.pdb. Recompiling a single file shows the identical path, but slightly different binary data.
So, the answer is, the project's Program Database File Name property was set to $(TargetDir)$(TargetName).pdb, when it should have been $(IntDir)%(Filename).pdb. Following this change, .pdb files appeared with each object file (i e, separate file in the same directory), and the build completes without any LNK4099 errors.
This solves the original problem. I am wondering, though, if there is a way to have the PDB information added to a single output file, rather than producing a separate PDB file for each object file.
I have received a crash dump file from a customer, and I am attempting to debug the dump file using VS2010. I have retrieved the source code we used for the build along with the generated symbol files, and I've provided VS with the path to the directory that contains them; everything seems to be in place. When I attempt to debug, however, VS reports that there is "no matching binary" for the .exe or any of the .dll's contained in that directory. I've used chkmatch to compare a few of the binaries and symbol files, and it reports that they match, time stamp and signature and all. I've searched around online and no one seems to have had as much trouble with this as I am encountering, and all the solutions I've found are things that I've already done.
Hopefully I'm overlooking something simple.
What am I missing? How can I ensure that VS matches the binaries to the symbol files?
You can use WinDbg
Download the tool (http://msdn.microsoft.com/en-us/windows/hardware/gg463016, you can use version 6.11.1.404 in order to avoid downloading the whole SDK) and install it.
start WinDbg and load the dump file (File > Open crash Dump).
specify the location of the symbols (File > Symbol File Path).
activate traces on symbols loading (!sym noisy at the command prompt)
unload all symbols (.reload /u at the command prompt).
load symbols for a module you have problem with (ld at the command prompt).
the engine will tell you where it loads symbols and why it doesn't match.
you can also have detailed informations on the module contained in the dump file (lm vm ), cinluding the timestamp of the module.
Hope this helps.
Another thing to check: I had problems if binaries and symbols for other build versions of the same product were in the symbol path.
VS (I'm using 2015) doesn't seem to search all the paths and choose the right files - I had to add symbol paths specific to the crash dump I'm debugging and move these "up" in the symbol path list for them to be found first and used.
I have heard using PDB files can help diagnose where a crash occurred.
My basic understanding is that you give Visual studio the source file, the pdb file and the crash information (from Dr Watson?)
Can someone please explain how it all works / what is involved?
(Thank you!)
PDB files map an assembly's MSIL to the original source lines. This means that if you put the PDB that was compiled with the assembly in the same directory as the assembly, your exception stack traces will have the names and lines of the positions in the original source files. Without the PDB file, you will only see the name of the class and method for each level of the stack trace.
PDB files are generated when you build your project. They contain information relating to the built binaries which Visual Studio can interpret.
When a program crashes and it generates a crash report, Visual Studio is able to take that report and link it back to the source code via the PDB file for the application. PDB files must be built from the same binary that generated the crash report!
There are some issues that we have encountered over time.
The machine that is debugging the crash report needs to have the source on the same path as the machine that built the binary.
Release builds often optimize to the extent where you cannot view the state of object member variables
If anyone knows how to defeat the former, I would be grateful for some input.
You should look into setting up a symbol server and indexing the PDB files to your source code control system. I just recently went through this process for our product and it works very well. You don't have to be concerned about making PDB files available with the binaries, nor how to get the appropriate source code when debugging dump files.
John Robbins' book: http://www.amazon.com/Debugging-Microsoft-NET-2-0-Applications/dp/0735622027/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1222366012&sr=8-1
Look here for some sample code for generating minidumps (which don't have to be restricted to post-crash analysis -- you can generate them at any point in your code without crashing): http://www.codeproject.com/KB/debug/postmortemdebug_standalone1.aspx