I am trying to debug a minidump crash dump. How can I get to know the OS so that I may download related Microsoft Symbols?
I am using Visual studio and windbg.
The dump contains enough information so that the symbol server client will load download and use the symbols that match the executable and dlls.
I.e. define environment variable _NT_SYMBOL_PATH to something like:
symsrv*symsrv.dll*h:\Symbols*http://referencesource.microsoft.com/symbols*http://msdl.microsoft.com/download/symbols
where h:\symbols is a writeable folder. The debugging tools will do the rest.
Note:
You need to do it this way because patches, including service packs, also change the version of the symbols.
The first download location is used for .NET with source server, if not working with .NET that can be removed.
vertarget will tell you the target machine of the minidump.
I haven't seen that syntax for the MS symbol server before, I usually just do:
.symfix h:\symbols
.reload
From within the debugging session. It automatically sets your symbol search path to point to the symbol server so you don't have to worry about remembering the path.
-scott
Related
My scenario is I'm supporting a VB6 app at the place I work and in the last few weeks it has started crashing more often than it ever used to. It uses both a local Access MDB database and a remote SQL Server DB for different types of storage. The good news is we are writing a replacement app, the band news I need to support this one in the meantime and the vendor is long gone from this world.
What are some ways I could try and diagnose what is causing the crash? For example so far I've tried ODBC tracing (For the MDB component), SQL Profiler tracing and ProcMon on a client PC.
Is there anything else I could try to discover what the app was trying to do at the time of the crash?
You can also start in a debugger.
windbg or ntsd (ntsd is a console program and maybe installed). Both are also from Debugging Tools For Windows.
Download and install Debugging Tools for Windows
http://msdn.microsoft.com/en-us/windows/hardware/hh852363
Install the Windows SDK but just choose the debugging tools.
Create a folder called Symbols in C:\
Start Windbg. File menu - Symbol File Path and enter
srv*C:\symbols*http://msdl.microsoft.com/download/symbols
then
windbg -o -g -G c:\windows\system32\cmd.exe /k batfile.bat
You can press F12 to stop it and kb will show the call stack (g continues the program). If there's errors it will also stop and show them.
Type lm to list loaded modules, x *!* to list the symbols and bp symbolname to set a breakpoint
Use db address (as in db 01244 to see what's at that memory.
If programming in VB6 then this environmental variable link=/pdb:none stores the symbols in the dll rather than seperate files. Make sure you compile the program with No Optimisations and tick the box for Create Symbolic Debug Info. Both on the Compile tab in the Project's Properties.
I have recently started windows driver development. I am wondering how does it actually debug my driver. The setup I have is --> Win7 as host, XP as guest on VMware, and I am debugging through serial port.
The research I have done:
I found only this link saying very few things that I am talking about.
I already know how debugger works on single OS, in that case debugger is also on the same OS, so it knows which process is running. That is understandable. But here, debugger is on entirely different OS, an entirely different environment. I just say file->open source files and I AM able to put breakpoints!! Moreover when I load driver, it actually breaks there. I mean why../How? How does XP's kernel comes to know(drivers are extension to kernel, atleast WDM, don't know about WDK) that there is source code of this driver? and that also outside its control(environment)? I mean I can have 10 files open with breakpoint in them, but it works beautifully, I am not able to fail/fool it.
So what I am thinking is like, whenever we add source to windbg on Win7, it creates the binary from that source, and whenever XP is going to load any binary, it checks if this is the binary that windbg is waiting for. what is confusing in above link is, Vikrant is saying that debugger asks kernel(XP) that it is willing to debug a process --> Bus HELLO... process is running on XP, and windbg on Win7 and does not know name or id of process. It has source code, but consider a case where there is a driver which is build out of 300 files, and just one, probably simplest file is open in windbg, how it matched that this source code is of the driver being run?
#Kjelll answer is correct. Here is the full scenario, including explanation to your comment:
PDB files have line information. This is a mapping from each (file,line) location to address (RVA - relative virtual address).
When you set a break point on a source file, WinDBG checks whether this source file correspond to a current address. If it is - it sets the breakpoint. Otherwise, it becomes a "future breakpoint" (not sure whether Microsoft uses this terminology).
When a new binary is a loaded, the agent on the client communicates with the host, informs it about the binary. At this point - WinDBG will try to allocate a PDB file.
WinDBG will start at the PDB location embedded within the file. You can see this value by using this command line: windbg -dump -pdbpath xxx.sys. This should explain how WinDBG will find a symbol file even if not on the .sympathy path (that I believe answers your comment to Kjell).
WinDBG will then search at the .sympathy.
Once symbol is find, it will look at all future breakpoint, and if applicable will set an actual breakpoint.
I think your link explain your question pretty well, but you have probably not realized what the mechanism of the pdb do for the debugger. The windbg on your host OS uses the pdb file to translate line nubers in the source files to addresses in your guest OS (xp) . Then the the debugger agent uses this address to set break points (Int 3) in the guest OS.This is much in the same way as a local debugger do to a local process.
I'm pretty new to WinDbg and I'm trying to find a bug which has my application hanging for no appearent reason. I'm not sure I'm doing things right, but I understand that I need both symbols for the system dlls aswell as the .exe I'm debugging. Thus, I set up my symbol path like this:
srv*c:\websymbols*http://msdl.microsoft.com/download/symbols;S:\MY\PATH
The second path pointing to a folder where I placed the .pdb that was generated by VS. I'm positive that's the correct .pdb file, but it was built on a different architecture (not sure if this is an issue). I would like to see a complete stack trace for a start, so I ran !analyze-v. The output looks like this. As you can see it lists APPLICATION_HANG_WRONG_SYMBOLS as primary problem. So I ran .reload /f, giving me this output. I have no symbols for dnAnalytics or Vertec.Interop, so these errors make sense, but there are some missing checksums and iphlpapi.pdb was not found.
So my questions would be: Why does WinDBG lists wrong symbols as primary problem, even though I'm positive I do have the right .pdb file available? (I'm running WinDBG on the same machine the dump was generated). To what extent can I trust the stack trace even thought my symbols are wrong, appearently? Does anyone see a obvious problem that could cause a hang in my application from the stack trace already? Any pointers appreciated!
The "wrong symbols" here is probably because you are using 64-bit CLR with version less than 4.0 and the !analyze extension has some trouble decoding the mixed native / managed stack.
Why is WinDBG looking for the BJM.exe
symbols on the microsoft server in
this case?
This is because you put the symbol server before you local path in the symbol path. Windbg does not know which module is yours and which is Microsoft's. It just looks for the PDB file for the module in the order speicified by the symbol path.
To what extent can I trust the stack trace even thought my symbols are wrong, appearently?
Stacks on x64 are very reliable since the stack-walk does not require symbols. Symbols are reliable (that is you do not have wrong symbols) unless you forced windbg to ignore wrong timestamp / checksum with .reload /f /i
In some cases the address -> symbol may seem wrong. This is usually due to small functions that have the same code (very common in C++ code if the functions are virtual or the code is not optimized)
Try using !sym noisy to get some more information about what it's actually looking at (docs). !itoldyouso may also be useful here (link)
What is the need/use of 'symbols' in the Microsoft debugger?
I spent some time trying to figure out the debugger a while back and never was able to get it making any sense (I was trying to debug a server hang...). Part of my problem was not having the proper 'symbols'.
What are they? And why would I need them? Aren't I just looking for text?
Are there any better links out there to using it than How to solve Windows system crashes in minutes ?
You need symbols in order to translate addresses into meaningful names. For example, you have locations on your stack at every function call:
0x00003791
0x00004a42
Symbols allows the debugger to map these addresses to methods
0x00003791 myprog!methodnamea
0x00004a42 myprog!methodnameb
When you build a debug version of a program, the compiler emits symbols with the extension .PDB. It also contains line information so you can do source code debugging, etc..
You need to set your symbol search path correctly for the debugger to pick this up. IN the command window you can do
.sympath c:\symbols;c:\temp\symbols
in order to have it search for the .PDB in these directories. It will also look in the same directory that the executable is ran from.
It also might be helpful to use the Microsoft public symbols server so that you can resolve OS binaries such as NTDLL, GDI, etc.. with this path at the beginning:
.sympath SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols;c:\symbols
You will need to create c:\websymbols first.
On the Windows binary architecture, the information needed for debugging (function names, file and line numbers, etc.) aren't present in the binary itself. Rather, they're collected into a PDB file (Program DataBase, file extension .pdb), which the debugger uses to correlate binary instructions with the sorts of information you probably use while debugging.
So in order to debug a server hang, you need the PDB file both for the server application itself, and optionally for the Windows binaries that your server is calling into.
As a general note, my experience with WinDbg is that it was much, much harder to learn how to use compared to GDB, but that it had much greater power once you understood how to use it. (The opposite of the usual case with Windows/Linux tools, interestingly.)
If you just have the binary file, the only info you can typically get is the stack trace, and maybe the binary or IL(in .NET) instructions. Having the symbols lets you actually match that binary/IL instruction up with a corresponding line in the source code. If you have the source code, it also lets you hook up the debugger in Visual Studio and step through the source code.
Within my application, I use the MiniDumpWriteDump function (see dbghelp.dll) to write a crash dump file whenever my application crashes.
I also use a symbol server to store all my executables and pdb files, so that whenever a customer sends me a crash-dump file, the debugger automatically picks up the correct version of the executable and the debug information.
I also store Windows DLL's (ntdll.dll, kernel32.dll, ...) and their debug information in the symbol server (using SymChk). The debug information is fetched from Microsoft's public symbol server.
Most of the time this works perfect, except when:
a customer crashes in one of the Windows DLL's
and the customer uses DLL's that I haven't put in the symbol server
This is because it is quite undoable to store every flavor of every Windows DLL in the Symbol Server (especially with the weekly patches).
So, if a customer crashes in, let's say, version 5.2.123.456 of NTDLL.DLL, and I didn't put this exact version of the DLL in my Symbol Server, then I'm stuck. Even Microsoft's public symbol server doesn't help because it only provides the debug information, not the DLL's itself.
My current solution is to ask the customer for his DLL's, but that's not always easy. Therefore I'm looking for a better solution.
Is there a way to get the debugger showing a correct call stack, or loading the debug information of a specific DLL, even if you don't have the exact version of the DLL?
Alternatively, is there a way to get all versions of all (or the important) Windows DLL's (from Microsoft)?
EDIT:
In the mean time I found a really easy way to solve this problem.
With the utility ModuleRescue (see http://www.debuginfo.com/tools/modulerescue.html) you can generate dummy DLL's from a minidump file. With these dummy DLL's, the debugger is satisfied, and correctly starts loading the debug symbols from the Microsoft servers.
It is possible to relax WinDbg's symbol resolution; see my answer to a similar question. On the other hand, the solution that I propose here relies on the fact that the DLLs are identical other than having different GUIDs identifying their debug symbols. A different version of a DLL is likely going to have a different binary, so the symbols are probably not going to match properly even if you can get them to load.
I'm pretty sure Microsoft's symbol server also provides the binaries. I'm looking in my store and I see a ton of Microsoft .dll files. I have my _NT_SYMBOL_PATH defined as
SRV*F:\Symbols\Microsoft*http://msdl.microsoft.com/download/symbols
This way it will search my local store first before trying to copy them down from Microsoft's public server.
You can have multiple symbol servers in your symbol path. So simply set up the symbol path to point to your own server for your own private modules, and to the public MS server for OS modules, see Symbol Path:
This can be easily combined with the
Microsoft public symbol store by using
the following initial setting:
_NT_SYMBOL_PATH=srv*c:\mysymbols*http://msdl.microsoft.com/download/symbols;cache*c:\mysymbols
The Microsoft Public Symbol Store is documented as http://msdl.microsoft.com/download/symbols.
? What part isn't working?
I've never been in your situation, exactly, but I would expect the debugger to give you the correct portion of the call stack that was in your code, up to the call into the dark dll. Of course, from there down to the actual crash symbols would be unavailable, but can't you see which NTDLL API was being called and which arguments were passed to that call?
You don't say which tool you're using for minidump debugging: WinDBG or VS.