I have been working on an application(Windows) on Visual Studio 2010 and encountered a crash. So for the same i asked for help from an expert and he asked me to submit a symbolized call stack of crash. Can anyone explain me what exactly is this symbolized call stack and how to get that from VS. The application i am building is using CEF(Chromium embedded framework). I have downloaded binaries and modifying it to my needs.
I think he is asking for a call stack with symbols available (as opposed to just call offsets like this - libcef.dll!11357796() Unknown).
Assuming you are developing with a binary distribution of CEF, go back to https://cefbuilds.com/ and find the distribution you used and the exact build number. At the end of the row you will see an additional link for Debug or Release Symbols. Download these and unzip them if zipped.
Now go back to your Visual Studio project, run until you hit your crash. Double click one of the unresolved symbols like "libcef.dll!11357796() Unknown". You will get a dialog asking you to locate the debug symbol file. Navigate to where you extracted the pdb files in the previous step, and it should find the symbols. It will then try to load the source file, which you don't have unless you downloaded CEF and Chromium sources, but you don't need that for a symbolic stack track. Just cancel and in your call stack you should now see symbols for all of libcef. Copy and paste and pass on to the person helping you.
It's one that has function names (and ideally filenames and line numbers) rather than plain hex addresses.
If you have debug symbols for your build, the call stack you get in the Visual Studio debugger's Call Stack window should be symbolized.
Related
I have a project in VS2012 where debugging is not available; if I set a breakpoint, it tells me that
"The breakpoint will not currently be hit. No symbols have been loaded for this document".
In the properties of the project, Generate debug info is set to full and optimizations are disabled (so the symbol files should be generated automatically...).
When inspecting the modules while in debug mode, the paths of the dlls are in the Temporary ASP.NET files folder, but no pdb files are to be found in that location. To be more specific; In the above mentioned folder, there are three subfolders (each named a seemingly random string of hex), and in one of these folders the dlls and pdbs are paired nicely for each module, however in the other two folders this is not the case, as only the dlls are present. Unfortunately VS is looking in one of the latter folders, thus not being able to find the symbol files.
Whenever I try to load in a symbol file manually for a dll, it tells me that symbol file isn't the right one; I assume it has to come from the exact same build as the dll. So I know where to put them, but I don't know where they are...
Since this folder is hidden (it's located in my AppData folder) and named "temporary...", I assume the files are loaded in automatically and that I shouldn't mess around with it.
So what do I do?
I know this is an extremely vague question, but I'm very ignorant when it comes to stuff like this, so some general guidance would be very much appreciated! Otherwise feel free to ask further elaborating questions - I'll do my very best to answer them.
P.S. I'm aware that very similar questions have been asked on here multiple times, but browsing through a lot of them, I still haven't found a solution that works for me, so I thought a new question where I can give you the exact details of my situation was appropriate - let me know if that is not the case, and I'll make sure to delete this post :)
1)Please check that which pdb file or symbol file was not really loaded, you could visit the debug Modules windows and find the information. Or you could visit the Output window, it would show you which pdb file was not found.
(2)Enable the Microsoft Symbol Servers under TOOLS->Options->Debugging->Symbols.
(3) Clean and Rebuild the solution, and make sure no compiled error, and then re-debug your app. Of course, you could visit the cache folder for the symbol servers, check that whether it has the pdb file which was not found before.
(4) If you find the pdb file, as Hans Passant's suggestion, you could also load the symbols manually in the Modules window:
1.Right-click in the debug Modules window, select Load Symbols From, then click Symbol Path or Microsoft Symbol Servers.
If the debugger doesn’t find the symbols, it would show the Find Symbols dialog.
2.In the Find Symbols dialog, locate the symbols file that you want to load.
The symbols file is a .pdb file or a .dbg file.
I was running a unit-test in Visual Studio today using FakeItEasy. I was offline and found the following symbol-loading to be happening and taking a long time:
My question is, where does the path Z:\Builds\work\... come from and why is Visual Studio trying to load symbols from that path. Could it be that this path corresponds to the CI that the binaries were built on? If so, is it a thing that the maintainer of the library should fix, or something that I must locally configure? I am using the FakeItEasy 1.25.2 binaries that I fetched via NuGet.
I am aware of the fact that you can disable symbol loading (e.g. see this question), but actually I want the symbols to be loaded if possible.
Yes, Z:\Builds\work\… is the path from which TeamCity builds FakeItEasy.
I'm not a big symbol user, so am not sure what you want "fixed". Why are you loading the symbols, and what behaviour would you expect in this case?
If we push the symbols to SymbolSource.org you'd still need to be online to access them, no?
Can you give an example of a NuGet package that behaves how you'd like?
How does it behave in your situation?
PDBs can be built for debug configuration and release configuration and it's typically a good idea to keep them for debugging purposes. FakeItEasy or any other DLL or EXE, contains the full path to the PDB file where it was located during compile time. If that path is part in a DLL (or EXE), Visual Studio will try to load symbols from there.
To see that information, get DebugDir and run debugdir <path to>\FakeItEasy.dll. Or, in any hex editor, search for pdb.
You'll find the full path of the PDB along with some other information. Since it wasn't you who built the DLL, the PDB is not present on your disk and you'd need to download it from a symbol server.
The Sourforge clone of DebugDir contains supports a command line parameter clean which can remove debug information. If you want to get rid of Visual Studio accessing the non-existing Z: drive, you can remove the path to the PDB file.
I have received a crash dump file from a customer, and I am attempting to debug the dump file using VS2010. I have retrieved the source code we used for the build along with the generated symbol files, and I've provided VS with the path to the directory that contains them; everything seems to be in place. When I attempt to debug, however, VS reports that there is "no matching binary" for the .exe or any of the .dll's contained in that directory. I've used chkmatch to compare a few of the binaries and symbol files, and it reports that they match, time stamp and signature and all. I've searched around online and no one seems to have had as much trouble with this as I am encountering, and all the solutions I've found are things that I've already done.
Hopefully I'm overlooking something simple.
What am I missing? How can I ensure that VS matches the binaries to the symbol files?
You can use WinDbg
Download the tool (http://msdn.microsoft.com/en-us/windows/hardware/gg463016, you can use version 6.11.1.404 in order to avoid downloading the whole SDK) and install it.
start WinDbg and load the dump file (File > Open crash Dump).
specify the location of the symbols (File > Symbol File Path).
activate traces on symbols loading (!sym noisy at the command prompt)
unload all symbols (.reload /u at the command prompt).
load symbols for a module you have problem with (ld at the command prompt).
the engine will tell you where it loads symbols and why it doesn't match.
you can also have detailed informations on the module contained in the dump file (lm vm ), cinluding the timestamp of the module.
Hope this helps.
Another thing to check: I had problems if binaries and symbols for other build versions of the same product were in the symbol path.
VS (I'm using 2015) doesn't seem to search all the paths and choose the right files - I had to add symbol paths specific to the crash dump I'm debugging and move these "up" in the symbol path list for them to be found first and used.
When using WinDbg, where should the private symbol files (pdb?) be placed?
My situation is: I have a DLL which I want to debug. I have the source code and symbol files for this DLL. This DLL is called by another DLL (which I don't have symbols or source for) which, in turn, is called by an EXE (which I also don't have symbols or source for).
My problem is that I am getting a warning that says
*** WARNING: Unable to verify checksum for C:\TheProgram\SomeSubfolder\AnotherSubfolder\MyDll.dll
This warning I think is the reason why I am getting the following type of messages in the call stack:
MyDll!AClass::AFunction+SomeHexAddress
My file structure looks something like this:
The exe: C:\TheProgram\program.exe
The calling dll: C\TheProgram\SomeSubfolder\caller.???
My DLL that I want to debug: C:\TheProgram\SomeSubfolder\AnotherSubfolder\MyDll.dll
Note: I set Symbol File path and the Source file path to where the debug DLL was generated, in my workspace on a different drive from the exe.. But I did copy the pdb + map files and put it on the dll that I wanted to debug..
Sorry for the late reply.
In your post you mention that you are seeing the following error message.
*** WARNING: Unable to verify checksum for C:\TheProgram\SomeSubfolder\AnotherSubfolder\MyDll.dll
You also ask the question, "where do I put my symbols for my DLL in the symbol path?"
Here is a response for the first problem:
Steps to identify mismatched symbols.
!sym noisy
.reload
x MyDll!*class*
*This reloads your dll, alternatively you can type kb to display the call stack of the DLL which should load it as well.
!sym quiet
*Reset's back to original quiet symbol loading
Also you can run
0:001> lmv m myDll *(and examine the Checksum)
Note: If you have a checksum, then Windbg can match the checksum of the DLL against the checksum of the PDB. Every development environment has a different way to generate a checksum.
Here is the response for the questions about where to put the PDBs
If you have MyDll.pdb added to a symbol store then you can use the following syntax
.sympath SRV*c:\symcache*http://msdl.microsoft.com/download/symbols
As Roger has suggested above...
However if you just have the PDB locally, you may want to put the path to the PDB first before going out to the symbol server like this
.sympath C:\TheProgram\SomeSubfolder\AnotherSubfolder\;SRV*c:\symcache*http://msdl.microsoft.com/download/symbols
This way Windbg should look local to your SomSubFolder dir before trying to use the Symbols Server cache.
Thanks,
Aaron
It does not matter where you put private symbol files as long as you're able to tell the debugger where they are.
The warning you're seeing does not have any effect on the stack trace, but the fact you're missing symbols for caller.DLL and app.EXE does.
Configuring symbols in windbg (locally) is as simple as using:
.sympath[+] path_to_pdbs
*and
.symfix+ path_to_system_pdb_store
You seeing:
MyDll!AClass::AFunction+SomeHexAddress
actually means nothing as long as SomeHexAddress is reasonable (and provided that MyDll.pdb has been found and loaded!) - it looks like a proper call stack entry.
Now, my question would be, what is the problem that you're stuck with?
P.S. you don't need .map file with windbg.
As part of our build process, we copy the private PDB files and the released EXE/DLL files to a symbol server. At its simplest, this is just a UNC path, but you can configure it for access using HTTP.
To copy your output files, use the SYMSTORE.EXE program.
Then, configure your debugger (we use Visual Studio and WinDbg) to look in that path. For WinDbg, the simplest way to do this is to set an environment variable:
_NT_SYMBOL_PATH=
SRV*C:\WebSymbols*http://msdl.microsoft.com/download/symbols;
\\symsvr\Symbols
(that should all be on one line)
This configures WinDbg to look on the Microsoft Symbol Server (caching the files in C:\WebSymbols) and also to look in a local symbol store (\\symsvr\Symbols).
We also use the Source Server tools to store SVN details in the PDB file, meaning that we can get back to the exact source file used to build a particular release. Look in ...\Debugging Tools for Windows (x86)\srcsrv.
One option is to leave the symbol files where they are (i.e. in the build output folder) and then use -y WinDbg command line option to locate these files. Using this approach should guarantee that the symbol files are always be up to date.
From the Microsoft Help:
-y SymbolPath
Specifies the symbol search path. Separate multiple paths with a
semicolon (;). If the path contains spaces, it should be enclosed
in quotation marks. For details, and for other ways to change this
path, see Symbol Path.
As it turned out, my target machine - provisioned inside Visual Studio - did not get the latest build upon deploying the to it, hence a "driver.sys has mismatched symbols" error.
Basically deploying did not replace the driver with the modified version of it for me. Use devcon tool to properly install it and Windbg will be happy again.
Are there any VC++ settings I should know about to generate better PDB files that contain more information?
I have a crash dump analysis system in place based on the project crashrpt.
Also, my production build server has the source code installed on the D:\, but my development machine has the source code on the C:\. I entered the source path in the VC++ settings, but when looking through the call stack of a crash, it doesn't automatically jump to my source code. I believe if I had my dev machine's source code on the D:\ it would work.
"Are there any VC++ settings I should know about"
Make sure you turn off Frame pointer ommision. Larry osterman's blog has the historical details about fpo and the issues it causes with debugging.
Symbols are loaded successfully. It shows the callstack, but double clicking on an entry doesn't bring me to the source code.
What version of VS are you using? (Or are you using Windbg?) ... in VS it should defintely prompt for source the first time if it doesn't find the location. However it also keeps a list of source that was 'not found' so it doesn't ask you for it every time. Sometimes the don't look list is a pain ... to get the prompt back up you need to go to solution explorer/solution node/properties/debug properties and edit the file list in the lower pane.
Finally you might be using 'stripped symbols'. These are pdb files generated to provide debug info for walking the callstack past FPO, but with source locations stripped out (along with other data). The public symbols for windows OS components are stripped pdbs. For your own code these simply cause pain and are not worth it unless you are providing your pdbs to externals. How would you have one of these horrible stripped pdbs? You might have them if you use "binplace" with the -a command.
Good luck! A proper mini dump story is a godsend for production debugging.
If your build directly from your sourcecode management system, you should annotate your pdb files with the file origins. This allows you to automatically fetch the exact source files while debugging. (This is the same proces as used for retrieving the .Net framework sourcecode).
See http://msdn.microsoft.com/en-us/magazine/cc163563.aspx for more information. If you use subversion as your SCM you can check out the SourceServerSharp project.
You could trying using the MS-DOS subst command to assign your source code directory to the D: drive.
This is the procedure I used after some trouble similar to yours:
a) Copied to the production server all the EXE & DLL files that were built, each with its corresponding PDB to the same directory, started the system, and waited for the crash to happen.
b) Copied back all the EXE, DLL & PDB files to the development machine (to a temporary folder) along with the minidump (in the same folder). Used Visual Studio to load the minidump from that folder.
Since VS found the source files where they were originally compiled, it was always able to identify them and load them correctly. As with you, in the production machine the drive used was not C:, but in the development machine it was.
Two more tips:
One thing I did often was to copy an EXE/DLL rebuilt and forget to copy the new PDB. This ruined the debug cycle, VS would not be able to show me the call stack.
Sometimes, I got a call stack that didn't make sense in VS. After some headache, I discovered that windbg would always show me the correct stack, but VS often wouldn't. Don't know why.
In case anyone is interested, a co-worker replied to this question to me via email:
Artem wrote:
There is a flag to MiniDumpWriteDump()
that can do better crash dumps that
will allow seeing full program state,
with all global variables, etc. As for
call stacks, I doubt they can be
better because of optimizations...
unless you turn (maybe some)
optimizations off.
Also, I think disabling inline
functions and whole program
optimization will help quite a lot.
In fact, there are many dump types,
maybe you could choose one small
enough but still having more info
http://msdn.microsoft.com/en-us/library/ms680519(VS.85).aspx
Those types won't help with call stack
though, they only affect the amount of
variables you'll be able to see.
I noticed some of those dump types
aren't supported in dbghelp.dll
version 5.1 that we use. We could
update it to the newest, 6.9 version
though, I've just checked the EULA for
MS Debugging Tools -- the newest
dbghelp.dll is still ok to
redistribute.
Is Visual Studio prompting you for the path to the source file? If it isn't then it doesn't think it has symbols for the callstack. Setting the source path should work without having to map the exact original location.
You can tell if symbols are loaded by looking at the 'modules' window in Visual Studio.
Assuming you are building a PDB then I don't think there are any options that control the amount of information in the PDB directly. You can change the type of optimizations performed by the compiler to improve debuggabilty, but this will cost performance -- as your co-worker points out, disabling inline will help make things more obvious in the crash file, but will cost at runtime.
Depending on the nature of your application I would recommend working with full dump files if you can, they are bigger, but give you all the information about the process ... and how often does it crash anyway :)
Is Visual Studio prompting you for the
path to the source file?
No.
If it isn't then it doesn't think it has symbols
for the callstack. Setting the source
path should work without having to map
the exact original location.
Symbols are loaded successfully. It shows the callstack, but double clicking on an entry doesn't bring me to the source code. I can of course search in files for the line in question, but this is hard work :)