Is there a way to re-add debug information to an existing Windows executable for which I have symbols? - windows

Crash Dump Debugging
Suspension of Disbelief: I know your first instinct may be to announce that "It isn't possible!" since this isn't how MS suggests solving the problem and they don't make tools available to accomplish it (easily). Forget that hurdle for a moment and ask yourself, "Is there some way to do this?"
I have a Windows binary I support in production that was built with debug information (MSVC 14.1 /Zi) but the debugger reports "was not built with debug information." This not wholly true—I think it was stripped by some tool during a post-build/pre-release process that I do not control, because I do have a PDB file for it. I have a memory dump from a run (crash) of that executable. Visual Studio and WinDbg both claim that the module has "no debug info," though WinDbg will attempt to load symbols for it anyway, if it finds some that match.
DBGHELP: No debug info for MyDll.dll. Searching for dbg file
DBGHELP: r:\crashes\symbols\MyDll.dbg - file not found
DBGHELP: r:\crashes\symbols\dll\MyDll.dbg - path not found
DBGHELP: r:\crashes\symbols\symbols\dll\MyDll.dbg - path not found
DBGHELP: MyDll.dll missing debug info. Searching for pdb anyway
*** WARNING: Unable to verify checksum for MyDll.dll
DBGENG: MyDll.dll has mismatched symbols - type ".hh dbgerr003" for details
DBGHELP: MyDll - private symbols & lines
r:\crashes\symbols\MyDll.pdb - unmatched
Here's the result of !chksym MyDll in WinDbg
MyDll.dll
Timestamp: 5CF5ABF8
SizeOfImage: 2437000
pdb sig: 0
age: 0
Loaded pdb is r:\crashes\symbols\MyDll.pdb
MyDll.pdb
pdb sig: 944F882B-73AE-45D0-9043-44C899BE09C5
age: 1
sig MISMATCH: MyDll.pdb and MyDll.dll
Is there a way to modify the memory dump (or the binary—assuming I can reproduce the problem) so as to re-inject whatever is necessary for the debugger to acknowledge that the symbols file I have matches the binary?
If your answer is "Nope!" I would appreciate some references to support that conclusion.
In the end, I want to be able to make sense of the call stack. Looking at variables/memory would be cool, too, but the call stack itself would be a gold mine in terms of debugging this problem.
Is "debug info" in this regard just a PE header (i.e. revealed by dumpbin /HEADERS) or is it more? Can I inject this header back into the binary/module? Is there more information that the debugger needs besides the symbols file to make sense of function addresses?
The tools for resolving symbol mismatches seem to work by modifying the PDB file. In my case, that doesn't seem to work because the binary module has no PDB signature in it to match to.
References
Crash dump - WinDbg - force PDB files to match doesn't work? (StackOverflow)
CHKMATCH (debuginfo.com)
How to Inspect the Content of a Program Database (PDB) File (codeproject.com)
microsoft-pdb (microsoft.com)
Corollary Questions
This is a problem I would like to solve one way or another (debugging a crash dump from a run of this "stripped" executable). So, it's possible that I'm asking the wrong question. Is there another way to solve this problem?
Can I decode memory addresses "outside" the Visual Studio debugger using the data in the PDB file?
Is there some other way to have the debugger use the symbols in a PDB file as if there was a match (assuming the module "has no debug info")?

If you can patch the PDB signature GUID and the age field from the PDB back into the binary, that might be enough to convince the tools that you have the right symbols. It seems likely those fields (or the section they live in) were cleared by the stripping tool.
The LLVM project has some documentation on the PDB format. Specifically, this section has some brief mentions about how PDBs and binaries are matched by the debuggers. And this section may tell you how to find (or recreate) the PDB Guid and age fields in the binary.
There are also some flags in the PE header (like IMAGE_FILE_LINE_NUMS_STRIPPED) that signal the debugger that information has been stripped. You might have to reset those to convince the debugger to even try to look for the PDB.
I don't know what else the stripping tool may have removed. In 32-bit binaries, I believe the frame-pointer opimization (FPO) data that's required for reliable stack unwinding is included in both the executable and the PDB. If the FPO data was stripped from the executable, I'd expect the debugger to use the data from the PDB, but I've not confirmed that. If I had to guess, I'd expect Windbg is more likely to rely on the PDB than the VS debugger.
In x86_64 binaries, basic stack unwinding is simpler (e.g., you should almost always be able to get a stack trace without much unwinding data). But if you want to examine locals and arguments, you do need some unwinding data, and that's stored in the PDB.
Sorry I have only clues rather than answers. I hope this is helpful.
Update 1: From your ChkMatch link, I read:
ChkMatch is capable of making an executable and PDB file match if they have different signatures but the same age (see this article for more information about PDB signature and age). If the age differs, the tool cannot make the files match. Signature and age can be displayed using -c option.
In the display you quoted, the ages are different (0 in executable and 1 in PDB). This seems the likely cause of why chkmatch -m won't solve your problem.

Related

VS IDE and WinDbg says that it cannot find symbol file

So, I primarily use VS IDE for debugging. I got a dump file and tried to do a postmortem on it. All of the DLLs loaded their respective PDBs except one and I don't know exactly why. This information would be helpful in determining if the dump file got corrupted in some way or if the client has a corrupted DLL.
I have also tried to use WinDbg to debug this, which I have some but not a lot of experience with. I updated the symbol paths to the directory that has the PDBs of the proper build and some others that it also might match up with as well. I loaded up the dump file and that same DLL is not having a matching PDB file found.
So the question is, what prevents a particular PDB not match with a dump file and how can I find out what that information is?
Symbols have a hash and a timestamp. Both need to match in order to load the symbols. In WinDbg, there's an option to force loading symbols that don't match (.symopt+ 0x40). Visual Studio doesn't have such an option, so you need to use chkmatch to make symbols match. Note that this is a dangerous operation, because it modifies the PDB file. You should create a backup copy and delete the modified file after you're done.
If you can't figure out what executable exactly is in the dump file, try .writemem <FileName> <Range> with the starting address of the executable and its size. See also How to retrieve assembly from a raw memory dump?.
For checking a dump file for corruption, I only know about DumpChk, which comes with WinDbg. AFAIK, the file format does not allow detection of single byte corruptions or similar.
I updated the symbol paths to the directory that has the PDBs
You should set up a symbol server. With a symbol server, there's no need to look for symbols or configure directories.

windbg refuses to load symbols for dump (Unable to verify timestamp)

I open a particular crash dump in windbg, set the symbol path, load the exception record, and produce a stack walk with k. I get a wrong call stack, even though i know the symbols do match.
The output of lmi is full of lines like 00007ff7'6f3c0000 00007ff7'743f0000 WhateverModule T (private pdb symbols) WhateverModule.dll, note the T there. According to the documentation, it means:
The timestamp is missing, not accessible, or equal to zero.
This checks out with what is described in this answer, indeed i did not specify a binary path. Even though the dump seems to contain all the timestamps for all the relevant modules (for instance, lmDvmWhateverModule outputs
[...]
Timestamp: Mon Jun 15 13:18:45 2015 (557EB495)
CheckSum: 04F8AF3B
[...]
) involved in the stacktrace, the debugger still complains. If i feed it a correct binary search path, then i get a correct callstack.
My questions are:
Even though the timestamps are right there in the dump, windbg complains about them missing. Why?
I thought PDBs are matched with binaries comparing a GUID and a so-called age. Why does windbg refuse to load symbols blaming missing information that it actually has, and does not need? :)
My windbg version is 10.0.10240.9
I was under the assumption that unwind information is present in PDB files. I was wrong, it isn't (by default). However, it turns out there is a linker switch which instructs the linker to copy unwind data into PDBs: /DEBUGTYPE:PDATA
All you need to do is add /DEBUGTYPE:CV,PDATA to your linker switches (according to the documentation, /DEBUGTYPE:CV is the default for user mode programs compiled with /DEBUG).
After this, windbg has no problem doing stackwalks, even with binaries missing (made sure with !sym noisy and lmvm that binaries are indeed not found). Unfortunately, this doesn't work in Visual Studio (tried with 2015 Update 3).

Trouble getting xperfview to load symbols for DLL

I have been fighting tooth and nail with xperf to get symbols for a tool I'm profiling. My code that runs within the tool is split between the .exe and a .dll -- the important stuff to profile being in the .dll. I ran xperf:
xperf -on PROC_THREAD+LOADER+INTERRUPT+DPC+PROFILE -stackwalk profile
And then I ran my tool for a bit, and then
xperf -d profile.etl
Then I tried xperfview. I loaded up the profile, toggled "load symbols" on, and opened the summary table. No symbols at all -- literally module came up "unknown" in the function column. I've scoured other threads on this and here's what I've tried:
I set my environment variables, _NT_SYMBOL_PATH and _NT_SYMCACHE
I cleared out my symbol cache and run xperf -symbols -i profile_results.etl.
I copied over dbghelp.dll from a recent version of Windows Debugging Tools and repeated the above.
After doing all this I now get function names showing up properly for most of the modules that are not my own code, but I can't get my dll to show up. The dll is being compiled in release mode (with optimization) but I set the Visual Studio project specifically to create a pdb, I've verified that the pdb exists and that it is within a directory on my _NT_SYMBOL_PATH. Does anyone know how I can fix this, or at least debug it further?
You can set some environment variables to enable diagnostic logging during symbol loading:
DBGHELP_DBGOUT = 1
DBGHELP_LOG = C:\dbghelp.log
I just encountered the same problem... tried all the same steps... browsed all the (apparently) similar advice...
Additionally, I tried launching symchk using the same dbghelp.dll/symsrv.dll DLLs I had copied into my WPA 'bin' folder, to make sure that my PDB is locatable. (still thinking I'm going crazy...)
I should note: my _NT_SYMBOL_PATH value contained servers with lcl cache & straight up local locations: _NT_SYMBOL_PATH=srv*D:\SymbolCache*http://msdl.microsoft.com/download/symbols;D:\GitHub\....
Then it dawned on me that my DLL, used by my "partner" EXE, is loaded dynamically via LoadLibrary()/GetProcAddress() ... could this be an issue for XPerf ?????
I hesitated even trying this...
I added a useless export in my DLL, and I invoke it directly in the EXE (to trigger an Import Table entry for my DLL) So now the EXE depends on the DLL to even load.
Turns out...
.............then XPerf loaded all the symbols :).
Edit: I just found this URL on MSDN, where someone posted code back in '11 that demonstrates a similar (the same?) problem
EDIT:
I recently discussed this with a collegue, and learned that XPerf will properly "decide" to load symbols for DLLs loaded programmatically ... IF the DLL remains loaded until the termination of the process.
So, for DLLs that are Loaded and Unloaded during the execution, and are unloaded at termination... XPerf will skip the attempt to load those symbols.
I'm not sure if this helps, but in here is one more detail I came across today in addition to the Q&A at xperf can't load my DLL's symbols:
For me, xperfview doesn't like PDB files on mapped network drives: as I was running xperf and xperfview on a different machine from where the code was built, I was getting both executables and PDB files off a network share, which I mapped to a drive letter to recreate exactly the same absolute paths as on the build machine - no luck. Even adding the folder with the PDB files to the symbol path didn't help.
Everything worked as expected once I made sure the .pdb file was in a local folder.
Try using wpa instead of xperfview. It uses the same system for loading symbols that xperfview does but it also has a Diagnostic Console which lets you see symbol loading messages which can be helpful.
Also, you should tell us what you have _NT_SYMBOL_PATH set to. There are many ways that it can be incorrectly set.
Also, in _NT_SYMBOL_PATH you should specify a local cache for your PDB files -- you can then check there to see if your PDBs have been copied to the local cache.
You can also look in the SymCache Path (pointed to by _NT_SYMCACHE_PATH, defaults to c:\symcache) which is where the WPT .symcache files are stored. The PDB files are converted to this format and the .symcache files are what are ultimately loaded by WPA and xperfview.
For more information see:
http://randomascii.wordpress.com/2012/10/04/xperf-symbol-loading-pitfalls/

WinDbg symbol resolution

When using WinDbg, where should the private symbol files (pdb?) be placed?
My situation is: I have a DLL which I want to debug. I have the source code and symbol files for this DLL. This DLL is called by another DLL (which I don't have symbols or source for) which, in turn, is called by an EXE (which I also don't have symbols or source for).
My problem is that I am getting a warning that says
*** WARNING: Unable to verify checksum for C:\TheProgram\SomeSubfolder\AnotherSubfolder\MyDll.dll
This warning I think is the reason why I am getting the following type of messages in the call stack:
MyDll!AClass::AFunction+SomeHexAddress
My file structure looks something like this:
The exe: C:\TheProgram\program.exe
The calling dll: C\TheProgram\SomeSubfolder\caller.???
My DLL that I want to debug: C:\TheProgram\SomeSubfolder\AnotherSubfolder\MyDll.dll
Note: I set Symbol File path and the Source file path to where the debug DLL was generated, in my workspace on a different drive from the exe.. But I did copy the pdb + map files and put it on the dll that I wanted to debug..
Sorry for the late reply.
In your post you mention that you are seeing the following error message.
*** WARNING: Unable to verify checksum for C:\TheProgram\SomeSubfolder\AnotherSubfolder\MyDll.dll
You also ask the question, "where do I put my symbols for my DLL in the symbol path?"
Here is a response for the first problem:
Steps to identify mismatched symbols.
!sym noisy
.reload
x MyDll!*class*
*This reloads your dll, alternatively you can type kb to display the call stack of the DLL which should load it as well.
!sym quiet
*Reset's back to original quiet symbol loading
Also you can run
0:001> lmv m myDll *(and examine the Checksum)
Note: If you have a checksum, then Windbg can match the checksum of the DLL against the checksum of the PDB. Every development environment has a different way to generate a checksum.
Here is the response for the questions about where to put the PDBs
If you have MyDll.pdb added to a symbol store then you can use the following syntax
.sympath SRV*c:\symcache*http://msdl.microsoft.com/download/symbols
As Roger has suggested above...
However if you just have the PDB locally, you may want to put the path to the PDB first before going out to the symbol server like this
.sympath C:\TheProgram\SomeSubfolder\AnotherSubfolder\;SRV*c:\symcache*http://msdl.microsoft.com/download/symbols
This way Windbg should look local to your SomSubFolder dir before trying to use the Symbols Server cache.
Thanks,
Aaron
It does not matter where you put private symbol files as long as you're able to tell the debugger where they are.
The warning you're seeing does not have any effect on the stack trace, but the fact you're missing symbols for caller.DLL and app.EXE does.
Configuring symbols in windbg (locally) is as simple as using:
.sympath[+] path_to_pdbs
*and
.symfix+ path_to_system_pdb_store
You seeing:
MyDll!AClass::AFunction+SomeHexAddress
actually means nothing as long as SomeHexAddress is reasonable (and provided that MyDll.pdb has been found and loaded!) - it looks like a proper call stack entry.
Now, my question would be, what is the problem that you're stuck with?
P.S. you don't need .map file with windbg.
As part of our build process, we copy the private PDB files and the released EXE/DLL files to a symbol server. At its simplest, this is just a UNC path, but you can configure it for access using HTTP.
To copy your output files, use the SYMSTORE.EXE program.
Then, configure your debugger (we use Visual Studio and WinDbg) to look in that path. For WinDbg, the simplest way to do this is to set an environment variable:
_NT_SYMBOL_PATH=
SRV*C:\WebSymbols*http://msdl.microsoft.com/download/symbols;
\\symsvr\Symbols
(that should all be on one line)
This configures WinDbg to look on the Microsoft Symbol Server (caching the files in C:\WebSymbols) and also to look in a local symbol store (\\symsvr\Symbols).
We also use the Source Server tools to store SVN details in the PDB file, meaning that we can get back to the exact source file used to build a particular release. Look in ...\Debugging Tools for Windows (x86)\srcsrv.
One option is to leave the symbol files where they are (i.e. in the build output folder) and then use -y WinDbg command line option to locate these files. Using this approach should guarantee that the symbol files are always be up to date.
From the Microsoft Help:
-y SymbolPath
Specifies the symbol search path. Separate multiple paths with a
semicolon (;). If the path contains spaces, it should be enclosed
in quotation marks. For details, and for other ways to change this
path, see Symbol Path.
As it turned out, my target machine - provisioned inside Visual Studio - did not get the latest build upon deploying the to it, hence a "driver.sys has mismatched symbols" error.
Basically deploying did not replace the driver with the modified version of it for me. Use devcon tool to properly install it and Windbg will be happy again.

WinDbg Dr. Watson minidump - requires pdb/dll originally built for installed version?

I have a mindmp file from a target's application crash. Is it possible for me to rebuild the dll/pdb files for a version of software and have windbg load symbols correctly?
My problem is that our pdb files are only kept for major releases (unfortunately). This is a daily build, which I can rebuild myself, but I'm getting tripped up on errors.
With !sym noisy on:
"image header does not match memory image header."
DBGENG: C:\...\XXX.dll image header does not match memory image header.
DBGENG: XXX.dll - Partial symbol image load missing image info
DBGHELP: Module is not fully loaded into memory.
DBGHELP: Searching for symbols using debugger-provided data.
DBGHELP: C:\...\XXX.pdb - mismatched pdb
Note I've build the pdb with the dll, they are from the same RELEASE directory (should I be building debug?)
Theses are release builds (as release builds are installed on the target and crashing) should I be somehow using the debug build dlls to get more symbol information?
The ChkMatch utility is designed for this exact scenario.
As long as you have the original .EXE, you can recompile the sources (with the same compiler and compiler settings) and patch the new .PDB to match the old .EXE.
In this example, OriginalExecutable.exe is the executable that no longer has a .PDB file, and RebuiltPDB.pdb is one that has been produced by rebuilding the original source.
chkmatch -m OriginalExecutable.exe RebuiltPDB.pdb
Now, as long as the two files have their original names, The debugger should accept them as a matching pair.
In my experience probably not.
If you have the exact build directory and build with the exact same compiler settings then this might work. You definitely will not be able to load symbols from a debug build against a release crash dump.
You will need to turn on the 'load anything' options: .symopt+0x40 to get windbg to ignore the timestamp differences.
if you still have the exact source code the image was compiled from, then rebuild it producing a new pdb file and then instruct WinDbg to forcibly load this pdb when you open the crash dump - it worked once in my practice.
PDB files are tied to their EXE files by a GUID and an "age" (it's a sequence number). These are embedded in the EXE, and into the PDB. The GUID is regenerated on each complete build, and the "age" is changed on each incremental build.
The debugger uses these to ensure that it's looking at the correct PDB for the EXE file.
I didn't know about the "chkmatch" tool mentioned by SteveMan, but I suspect that it works by patching up the GUID/age so that they match.
This is too late to help Doug, but for the sake of anyone who comes across this question, another thread (Is it possible to load mismatched symbols in Visual Studio?) pointed out a way to get WinDbg to accept mismatched .PDB files
.symopt_0x40

Resources