WinDbg symbol resolution - debugging

When using WinDbg, where should the private symbol files (pdb?) be placed?
My situation is: I have a DLL which I want to debug. I have the source code and symbol files for this DLL. This DLL is called by another DLL (which I don't have symbols or source for) which, in turn, is called by an EXE (which I also don't have symbols or source for).
My problem is that I am getting a warning that says
*** WARNING: Unable to verify checksum for C:\TheProgram\SomeSubfolder\AnotherSubfolder\MyDll.dll
This warning I think is the reason why I am getting the following type of messages in the call stack:
MyDll!AClass::AFunction+SomeHexAddress
My file structure looks something like this:
The exe: C:\TheProgram\program.exe
The calling dll: C\TheProgram\SomeSubfolder\caller.???
My DLL that I want to debug: C:\TheProgram\SomeSubfolder\AnotherSubfolder\MyDll.dll
Note: I set Symbol File path and the Source file path to where the debug DLL was generated, in my workspace on a different drive from the exe.. But I did copy the pdb + map files and put it on the dll that I wanted to debug..

Sorry for the late reply.
In your post you mention that you are seeing the following error message.
*** WARNING: Unable to verify checksum for C:\TheProgram\SomeSubfolder\AnotherSubfolder\MyDll.dll
You also ask the question, "where do I put my symbols for my DLL in the symbol path?"
Here is a response for the first problem:
Steps to identify mismatched symbols.
!sym noisy
.reload
x MyDll!*class*
*This reloads your dll, alternatively you can type kb to display the call stack of the DLL which should load it as well.
!sym quiet
*Reset's back to original quiet symbol loading
Also you can run
0:001> lmv m myDll *(and examine the Checksum)
Note: If you have a checksum, then Windbg can match the checksum of the DLL against the checksum of the PDB. Every development environment has a different way to generate a checksum.
Here is the response for the questions about where to put the PDBs
If you have MyDll.pdb added to a symbol store then you can use the following syntax
.sympath SRV*c:\symcache*http://msdl.microsoft.com/download/symbols
As Roger has suggested above...
However if you just have the PDB locally, you may want to put the path to the PDB first before going out to the symbol server like this
.sympath C:\TheProgram\SomeSubfolder\AnotherSubfolder\;SRV*c:\symcache*http://msdl.microsoft.com/download/symbols
This way Windbg should look local to your SomSubFolder dir before trying to use the Symbols Server cache.
Thanks,
Aaron

It does not matter where you put private symbol files as long as you're able to tell the debugger where they are.
The warning you're seeing does not have any effect on the stack trace, but the fact you're missing symbols for caller.DLL and app.EXE does.
Configuring symbols in windbg (locally) is as simple as using:
.sympath[+] path_to_pdbs
*and
.symfix+ path_to_system_pdb_store
You seeing:
MyDll!AClass::AFunction+SomeHexAddress
actually means nothing as long as SomeHexAddress is reasonable (and provided that MyDll.pdb has been found and loaded!) - it looks like a proper call stack entry.
Now, my question would be, what is the problem that you're stuck with?
P.S. you don't need .map file with windbg.

As part of our build process, we copy the private PDB files and the released EXE/DLL files to a symbol server. At its simplest, this is just a UNC path, but you can configure it for access using HTTP.
To copy your output files, use the SYMSTORE.EXE program.
Then, configure your debugger (we use Visual Studio and WinDbg) to look in that path. For WinDbg, the simplest way to do this is to set an environment variable:
_NT_SYMBOL_PATH=
SRV*C:\WebSymbols*http://msdl.microsoft.com/download/symbols;
\\symsvr\Symbols
(that should all be on one line)
This configures WinDbg to look on the Microsoft Symbol Server (caching the files in C:\WebSymbols) and also to look in a local symbol store (\\symsvr\Symbols).
We also use the Source Server tools to store SVN details in the PDB file, meaning that we can get back to the exact source file used to build a particular release. Look in ...\Debugging Tools for Windows (x86)\srcsrv.

One option is to leave the symbol files where they are (i.e. in the build output folder) and then use -y WinDbg command line option to locate these files. Using this approach should guarantee that the symbol files are always be up to date.
From the Microsoft Help:
-y SymbolPath
Specifies the symbol search path. Separate multiple paths with a
semicolon (;). If the path contains spaces, it should be enclosed
in quotation marks. For details, and for other ways to change this
path, see Symbol Path.

As it turned out, my target machine - provisioned inside Visual Studio - did not get the latest build upon deploying the to it, hence a "driver.sys has mismatched symbols" error.
Basically deploying did not replace the driver with the modified version of it for me. Use devcon tool to properly install it and Windbg will be happy again.

Related

VS IDE and WinDbg says that it cannot find symbol file

So, I primarily use VS IDE for debugging. I got a dump file and tried to do a postmortem on it. All of the DLLs loaded their respective PDBs except one and I don't know exactly why. This information would be helpful in determining if the dump file got corrupted in some way or if the client has a corrupted DLL.
I have also tried to use WinDbg to debug this, which I have some but not a lot of experience with. I updated the symbol paths to the directory that has the PDBs of the proper build and some others that it also might match up with as well. I loaded up the dump file and that same DLL is not having a matching PDB file found.
So the question is, what prevents a particular PDB not match with a dump file and how can I find out what that information is?
Symbols have a hash and a timestamp. Both need to match in order to load the symbols. In WinDbg, there's an option to force loading symbols that don't match (.symopt+ 0x40). Visual Studio doesn't have such an option, so you need to use chkmatch to make symbols match. Note that this is a dangerous operation, because it modifies the PDB file. You should create a backup copy and delete the modified file after you're done.
If you can't figure out what executable exactly is in the dump file, try .writemem <FileName> <Range> with the starting address of the executable and its size. See also How to retrieve assembly from a raw memory dump?.
For checking a dump file for corruption, I only know about DumpChk, which comes with WinDbg. AFAIK, the file format does not allow detection of single byte corruptions or similar.
I updated the symbol paths to the directory that has the PDBs
You should set up a symbol server. With a symbol server, there's no need to look for symbols or configure directories.

Symbol path for WinDbg

My pc underwent an automatic restart and when I checked event viewer, I found the bug check error. I found the location of the .dmp file created. I then downloaded WinDbg and understand the first thing I have to do is set the symbol path and then open crash dump and select the .dmp file. I found a list of the paths to use at https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/setting-symbol-and-source-paths-in-windbg but whenever I open crash dump and click the file, it gives me the error that regarding the symbols. I've tried a lot of the paths listed, but every time I get this message repeated:
Either you specified an unqualified symbol, or your debugger *
doesn't have full symbol information. Unqualified symbol
resolution is turned off by default. Please either specify a
fully qualified symbol module!symbolname, or enable resolution
of unqualified symbols by typing ".symopt- 100". Note that
enabling unqualified symbol resolution with network symbol
server shares in the symbol path may cause the debugger to
appear to hang for long periods of time when an incorrect
symbol name is typed or the network symbol server is down.
For some commands to work properly, your symbol path
must point to .pdb files that have full type information.
Certain .pdb files (such as the public OS symbols) do not
contain the required information. Contact the group that
provided you with these symbols if you need this command to
work.
* Type referenced: nt!_KPRCB
This leads me to the conclusion that I did not put the correct symbol path. I would appreciate any advice on which to choose or what else to do. I'm not using a separate system and I want to use the Microsoft server.
WinDbg has a workspace concept and when you load a dump, the workspace is created for that. To avoid it asking for symbol settings every time, please do the following:
Launch WinDbg
Do not open any dump
Go to the Symbol Settings and set a cache and server path like:
srv*c:\symcache*http://msdl.microsoft.com/download/symbols
On the file menu click "Save Workspace"
Close WinDbg
Next time, you do not have to set the symbol path for any dumps you open.
You should make sure your have an Internet connection, then do a
.symfix
.reload /f
!analyze -v
and check if the problems are gone. If not, it might happen that those symbols are simply not present.
If the problem is not gone, notify Andy Luhrs #aluhrs13 about the issue, make sure you include the relevant information. Potentially related: missing win32k.sys

Visual Studio tries to load FakeItEasy symbols from Z:\Builds\work

I was running a unit-test in Visual Studio today using FakeItEasy. I was offline and found the following symbol-loading to be happening and taking a long time:
My question is, where does the path Z:\Builds\work\... come from and why is Visual Studio trying to load symbols from that path. Could it be that this path corresponds to the CI that the binaries were built on? If so, is it a thing that the maintainer of the library should fix, or something that I must locally configure? I am using the FakeItEasy 1.25.2 binaries that I fetched via NuGet.
I am aware of the fact that you can disable symbol loading (e.g. see this question), but actually I want the symbols to be loaded if possible.
Yes, Z:\Builds\work\… is the path from which TeamCity builds FakeItEasy.
I'm not a big symbol user, so am not sure what you want "fixed". Why are you loading the symbols, and what behaviour would you expect in this case?
If we push the symbols to SymbolSource.org you'd still need to be online to access them, no?
Can you give an example of a NuGet package that behaves how you'd like?
How does it behave in your situation?
PDBs can be built for debug configuration and release configuration and it's typically a good idea to keep them for debugging purposes. FakeItEasy or any other DLL or EXE, contains the full path to the PDB file where it was located during compile time. If that path is part in a DLL (or EXE), Visual Studio will try to load symbols from there.
To see that information, get DebugDir and run debugdir <path to>\FakeItEasy.dll. Or, in any hex editor, search for pdb.
You'll find the full path of the PDB along with some other information. Since it wasn't you who built the DLL, the PDB is not present on your disk and you'd need to download it from a symbol server.
The Sourforge clone of DebugDir contains supports a command line parameter clean which can remove debug information. If you want to get rid of Visual Studio accessing the non-existing Z: drive, you can remove the path to the PDB file.

Debugging crash dump: binary and symbol files from same build event do not match according to VS2010

I have received a crash dump file from a customer, and I am attempting to debug the dump file using VS2010. I have retrieved the source code we used for the build along with the generated symbol files, and I've provided VS with the path to the directory that contains them; everything seems to be in place. When I attempt to debug, however, VS reports that there is "no matching binary" for the .exe or any of the .dll's contained in that directory. I've used chkmatch to compare a few of the binaries and symbol files, and it reports that they match, time stamp and signature and all. I've searched around online and no one seems to have had as much trouble with this as I am encountering, and all the solutions I've found are things that I've already done.
Hopefully I'm overlooking something simple.
What am I missing? How can I ensure that VS matches the binaries to the symbol files?
You can use WinDbg
Download the tool (http://msdn.microsoft.com/en-us/windows/hardware/gg463016, you can use version 6.11.1.404 in order to avoid downloading the whole SDK) and install it.
start WinDbg and load the dump file (File > Open crash Dump).
specify the location of the symbols (File > Symbol File Path).
activate traces on symbols loading (!sym noisy at the command prompt)
unload all symbols (.reload /u at the command prompt).
load symbols for a module you have problem with (ld at the command prompt).
the engine will tell you where it loads symbols and why it doesn't match.
you can also have detailed informations on the module contained in the dump file (lm vm ), cinluding the timestamp of the module.
Hope this helps.
Another thing to check: I had problems if binaries and symbols for other build versions of the same product were in the symbol path.
VS (I'm using 2015) doesn't seem to search all the paths and choose the right files - I had to add symbol paths specific to the crash dump I'm debugging and move these "up" in the symbol path list for them to be found first and used.

Trouble getting xperfview to load symbols for DLL

I have been fighting tooth and nail with xperf to get symbols for a tool I'm profiling. My code that runs within the tool is split between the .exe and a .dll -- the important stuff to profile being in the .dll. I ran xperf:
xperf -on PROC_THREAD+LOADER+INTERRUPT+DPC+PROFILE -stackwalk profile
And then I ran my tool for a bit, and then
xperf -d profile.etl
Then I tried xperfview. I loaded up the profile, toggled "load symbols" on, and opened the summary table. No symbols at all -- literally module came up "unknown" in the function column. I've scoured other threads on this and here's what I've tried:
I set my environment variables, _NT_SYMBOL_PATH and _NT_SYMCACHE
I cleared out my symbol cache and run xperf -symbols -i profile_results.etl.
I copied over dbghelp.dll from a recent version of Windows Debugging Tools and repeated the above.
After doing all this I now get function names showing up properly for most of the modules that are not my own code, but I can't get my dll to show up. The dll is being compiled in release mode (with optimization) but I set the Visual Studio project specifically to create a pdb, I've verified that the pdb exists and that it is within a directory on my _NT_SYMBOL_PATH. Does anyone know how I can fix this, or at least debug it further?
You can set some environment variables to enable diagnostic logging during symbol loading:
DBGHELP_DBGOUT = 1
DBGHELP_LOG = C:\dbghelp.log
I just encountered the same problem... tried all the same steps... browsed all the (apparently) similar advice...
Additionally, I tried launching symchk using the same dbghelp.dll/symsrv.dll DLLs I had copied into my WPA 'bin' folder, to make sure that my PDB is locatable. (still thinking I'm going crazy...)
I should note: my _NT_SYMBOL_PATH value contained servers with lcl cache & straight up local locations: _NT_SYMBOL_PATH=srv*D:\SymbolCache*http://msdl.microsoft.com/download/symbols;D:\GitHub\....
Then it dawned on me that my DLL, used by my "partner" EXE, is loaded dynamically via LoadLibrary()/GetProcAddress() ... could this be an issue for XPerf ?????
I hesitated even trying this...
I added a useless export in my DLL, and I invoke it directly in the EXE (to trigger an Import Table entry for my DLL) So now the EXE depends on the DLL to even load.
Turns out...
.............then XPerf loaded all the symbols :).
Edit: I just found this URL on MSDN, where someone posted code back in '11 that demonstrates a similar (the same?) problem
EDIT:
I recently discussed this with a collegue, and learned that XPerf will properly "decide" to load symbols for DLLs loaded programmatically ... IF the DLL remains loaded until the termination of the process.
So, for DLLs that are Loaded and Unloaded during the execution, and are unloaded at termination... XPerf will skip the attempt to load those symbols.
I'm not sure if this helps, but in here is one more detail I came across today in addition to the Q&A at xperf can't load my DLL's symbols:
For me, xperfview doesn't like PDB files on mapped network drives: as I was running xperf and xperfview on a different machine from where the code was built, I was getting both executables and PDB files off a network share, which I mapped to a drive letter to recreate exactly the same absolute paths as on the build machine - no luck. Even adding the folder with the PDB files to the symbol path didn't help.
Everything worked as expected once I made sure the .pdb file was in a local folder.
Try using wpa instead of xperfview. It uses the same system for loading symbols that xperfview does but it also has a Diagnostic Console which lets you see symbol loading messages which can be helpful.
Also, you should tell us what you have _NT_SYMBOL_PATH set to. There are many ways that it can be incorrectly set.
Also, in _NT_SYMBOL_PATH you should specify a local cache for your PDB files -- you can then check there to see if your PDBs have been copied to the local cache.
You can also look in the SymCache Path (pointed to by _NT_SYMCACHE_PATH, defaults to c:\symcache) which is where the WPT .symcache files are stored. The PDB files are converted to this format and the .symcache files are what are ultimately loaded by WPA and xperfview.
For more information see:
http://randomascii.wordpress.com/2012/10/04/xperf-symbol-loading-pitfalls/

Resources