When I do a dump file analysis, I setup symbol path to point to Microsoft symbol server. How does windbg knows that the dump file was generated under what OS and how does it downloads correct symbols for that OS?
To match symbols to binary, WinDbg looks at the thing called Debug Directories. Debug Directories are sections in PE modules (which is a file format used by Windows for all types of executables). Debug Directories simply contain links to types of debug information. If you type command in cmd window link /dump /headers <my_module_name>, it will print out something like this:
...
Debug Directories
Time Type Size RVA Pointer
-------- ------ -------- -------- --------
4CC78FB1 cv 22 00102588 101988 Format: RSDS, {30976E0B-FBF7-45EF-8608-99932F2B791F}, 2, ntdll.pdb
4CC78FB1 ( A) 4 00102584 101984 BB03197E
...
This is output for ntdll.dll. You can see that CV (for CodeView) debug information is contained in ntdll.pdb, and GUID of that PDB has to match the one in the link. That GUID is generated randomly for each module at build time.
Command !lmi in WinDbg will also dump this information, in different format.
What WinDbg does when loading symbols from symbol server is it sends request to symbol server to get file named 'ntdll.pdb' with matching GUID.
Related
I've used Ubuntu 20.04.3, and when I run
readelf --string-dump=.gnu_debuglink /usr/bin/cp
I got weird debug filename while expecting something like cp.debug
String dump of section '.gnu_debuglink':
[ 0] 674b3a5e9ca27e34cf3517aa997ba91ce6e0a0.debug
[ 31] k`-
This is the original Ubuntu image, no modification at all.
Is there any reason for this?
I got weird debug filename while expecting something like cp.debug
There is nothing weird about this name.
There are two common schemes for associating the program and debug info for that program -- name of the program, or its linker build-id.
You can find linker build-id with readelf -n /usr/bin/cp. Here is the output on my system:
Displaying notes found in: .note.gnu.build-id
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) Build ID: 2f6b630344b1b72875f756dce05a40186d18c6d8
Displaying notes found in: .note.ABI-tag
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag) OS: Linux, ABI: 3.2.0
Chances are, the linker build-id for your version of cp is 674b3a5e9ca27e34cf3517aa997ba91ce6e0a0.
Using linker build-id is especially convenient for programs which can appear under several different names (where using just the program name you would have to create several copies (or links) of the debug info).
Also, if the program is updated, its linker build-id will change, and so there is no chance that the "stale" debug info would be loaded by the debugger.
On my fedora box I have installed a lot of separate debug infos.
sudo dnf debuginfo-install <list of packets>
Now, if I debug some simple code it needs very long until some symbol is displayed or some values are printed. It is quite clear that is absolutly needed to evaluate all the installed symbol files to get all information.
But if I have a problem, say on a lib like goocanvas I only want to have my local debug smbols generated with my own compiled code with -g option and the only the debug infos for goocanvas libs.
How can that kind of selection be achieved? Only by renaming the folder of debug info files and generate a copy of needed ones? Maybe as a symlink? Or is there a common selection option anywhere?
You can skip all debug info from shared libraries and only load goocanvas lib symbols. Here is a sample of how to do it in gdb session:
[ ~]$ gdb -q /your/binary
(gdb) set auto-solib-add off
(gdb) start
Temporary breakpoint 1, 0x000055555564edd0 in main ()
(gdb) sharedlibrary goocanvas
From gdb doc:
If your program uses lots of shared libraries with debug info that
takes large amounts of memory, you can decrease the gdb memory
footprint by preventing it from automatically loading the symbols from
shared libraries. To that end, type set auto-solib-add off before
running the inferior, then load each library whose debug symbols you
do need with sharedlibrary regexp, where regexp is a regular
expression that matches the libraries whose symbols you want to be
loaded.
See also this related question: How to prevent GDB from loading debugging symbol for a (large) library?
I have a few Windows 7 machines that I am not able to read their memory dumps. I found something that I suspect may be related, but am not positive:
https://twitter.com/aionescu/status/634028737458114560
I also found this: http://support.microsoft.com/kb/2528507
However, the scenario message regarding wow64exts given in the doc is not seen in any of my dumps. I also cannot apply that hotfix at this time to test it. So I'm just looking for some more information or opinions.
I'm able to open any other OS dump as well as my own system's Windows 7 dump, but there are 2 other machines that run Win 7 and it's telling me I have the wrong kernel symbols.
I have tried clearing out my symbol cache, reinstalled the Windows SDK, and also tried to open the dumps on two other machines with the same result. If it matters, the crash is manually created using the scroll lock method.
Symbol path: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols;
Seeing these errors: followed by "Type referenced: nt!_KPRCB"
Does anyone know about the issue mentioned by Alex in the twitter link and if it's possibly related to what I'm seeing?
Update 2015-10-22:
With the Microsoft patch day (2015-10-13) and KB3088195, symbols are available again.
However, symbols for the broken version have not been provided, so below may still be useful.
Microsoft has already published "good" symbols for ntdll in the past, containing type information like _TEB or _KPRCB. Starting from mid of July 2015, Microsoft has still published symbols for ntdll, but not containing that information.
So it depends on the version of ntdll whether you get type information or not. Old dumps referencing an old version of ntdll will download old PDBs containing type information while new dumps reference new versions of ntdll and WinDbg (or any other debugger) downloads PDBs without type information.
Could Microsoft remove type information of "good" symbols retroactively, thus making them "bad"?
Yes. As described in this answer, there is a tool to remove type information from existing PDBs. Doing that and replacing the PDB would result in such an effect.
Can Microsoft publish the "good" version of those PDBs which are currently "bad"?
That's hard to tell, since we don't know whether Microsoft has kept a copy of the "good" version so they could replace the "bad" version on the symbol server with the "good" one. Rebuilding ntdll from the same source code and thus creating new PDBs sounds possible, but the PDB gets a new time stamp and checksum. This can potentially be corrected manually, especially be Microsoft, since they should have the knowledge about the PDB internal format, but IMHO it's unlikely they'll do that. Things may go wrong and MS will hardly have tests to guarantee the correctness of such a thing.
So what can I do?
IMHO you can do nothing to really correct the situation.
You could assume that the types in ntdll have not changed so much. This would allow you to take an older version of wntdll.pdb and the new version of ntdll.dll and apply ChkMatch -m to it. This will copy the timestamp and checksum from the DLL to the PDB. After you did that (in an empty folder), replace the existing wntdll.pdb in your symbols directory with the hacked one.
WinDbg walkthrough (with output shortened to relevant things). I am using the latest version of wntdll.pdb I could find on my PC.
WARNING: doing the following may fix the type information but will likely destroy the correctness of the callstacks. Since any changes in the implementation (which are likely for security fixes) will change the method offsets.
0:005> dt nt!_PEB
*************************************************************************
*** ***
*** Either you specified an unqualified symbol, or your debugger ***
...
*** Type referenced: nt!_PEB ***
*** ***
*************************************************************************
Symbol nt!_PEB not found.
0:005> lm m ntdll
start end module name
773f0000 77570000 ntdll (pdb symbols) e:\debug\symbols\wntdll.pdb\FA9C48F9C11D4E0894B8970DECD92C972\wntdll.pdb
0:005> .shell cmd /c copy C:\Windows\SysWOW64\ntdll.dll e:\debug\temp\ntdllhack\ntdll.dll
1 file(s) copied.
0:005> .shell cmd /c copy "E:\Windows SDk\8.0\Debuggers\x86\sym\wntdll.pdb\B081677DFC724CC4AC53992627BEEA242\wntdll.pdb" e:\debug\temp\ntdllhack\wntdll.pdb
1 file(s) copied.
0:005> .shell cmd /c E:\debug\temp\ntdllhack\chkmatch.exe -m E:\debug\temp\ntdllhack\ntdll.dll E:\debug\temp\ntdllhack\wntdll.pdb
...
Executable: E:\debug\temp\ntdllhack\ntdll.dll
Debug info file: E:\debug\temp\ntdllhack\wntdll.pdb
Executable:
TimeDateStamp: 55a69e20
Debug info: 2 ( CodeView )
TimeStamp: 55a68c18 Characteristics: 0 MajorVer: 0 MinorVer: 0
Size: 35 RVA: 000e63e0 FileOffset: 000d67e0
CodeView format: RSDS
Signature: {fa9c48f9-c11d-4e08-94b8-970decd92c97} Age: 2
PdbFile: wntdll.pdb
Debug info: 10 ( Unknown )
TimeStamp: 55a68c18 Characteristics: 0 MajorVer: 565 MinorVer: 6526
Size: 4 RVA: 000e63dc FileOffset: 000d67dc
Debug information file:
Format: PDB 7.00
Signature: {b081677d-fc72-4cc4-ac53-992627beea24} Age: 4
Writing to the debug information file...
Result: Success.
0:005> .shell cmd /c copy E:\debug\temp\ntdllhack\wntdll.pdb E:\debug\symbols\wntdll.pdb\FA9C48F9C11D4E0894B8970DECD92C972\wntdll.pdb
1 file(s) copied.
0:005> .reload
Reloading current modules
.............................
0:005> dt nt!_PEB
ntdll!_PEB
+0x000 InheritedAddressSpace : UChar
+0x001 ReadImageFileExecOptions : UChar
...
0:005> !heap -s
LFH Key : 0x219ab08b
Termination on corruption : DISABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-----------------------------------------------------------------------------
Virtual block: 00920000 - 00920000 (size 00000000)
Virtual block: 02c60000 - 02c60000 (size 00000000)
Virtual block: 02e10000 - 02e10000 (size 00000000)
...
Note: using ChkMatch like this has the benefit that you do not need to turn on .symopt- 100, since that option would affect all PDB files, and you would not find potential other symbol issues. If you don't mind using .symopt, you could simply copy an old wntdll.PDB over the new one.
The issue is now fixed according to Microsoft and Microsoft told me that you should clear your symbol cache to get the new PDBs, otherwise Windbg would use the old Symbols which miss the information.
I have a minidump from a customer. I want to find out the assembly versions of the loaded .NET dlls. I've already searched the internet for hours now, but cannot find a usable way.
I have windbg and have loaded SOS extension an have the needed clr.dll and mscordacwks.
using lm -v only shows the unmanaged dll's. I am sure I am overlooking something very simple.
WinDbg's lm v shows all DLLs, managed and unmanaged. On that level, an assembly is just a DLL, is just a module.
With WinDbg 6.3.9600, lm v even shows information whether a modules has a CLR header or not:
0:008> lmv m MyApp
start end module name
10310000 10574000 MyApp(deferred)
Image path: C:\...\MyApp.exe
Image name: MyApp.exe
Has CLR image header, track-debug-data flag not set
Timestamp: Wed May 21 16:34:02 2014 (537CB95A)
...
ProductVersion: 1.8.0.44
FileVersion: 1.8.0.44
...
However, the FileVersion and ProductVersion is still unmanaged information compiled into the native resource part of the DLL.
I suggest you use !SaveModule <start address> <filename> to save the file on disk and then use a reflection tool like dotPeek to open it and get the assembly information from there, which will give you the name, version, culture and public key token.
To save all modules, Naveen has written a script like this:
!for_each_module .if ($spat ("${##ImageName}","*.exe")) { !SaveModule ${##Base} c:\temp\${##ModuleName}.exe } .else { !SaveModule ${##Base} c:\temp\${##ModuleName}.dll }
Alright, after some more research I came to the conclusion that that kind of information is just not available for those dll's in a minidump. Anyway I was at least able to get some more information about the files, and maybe this is useful for someone else in the future.
You can get at least the metata information of the dll's, and maybe you can find something useful in there. Here's how to do that:
First get the domain:
!dumpdomain
You might end up with a large amount of listed assemblies. Now you can either look them up by hand using:
!dumpassembly ADDRESS
or you can just use the lazy way: Install the python extension for windbg (http://pykd.codeplex.com/) and use the following script for a quick overlook (it's maybe not the nicest script, but it's working and I didn't want to invest more time):
import pykd
def dump_assemblies():
assemblies = 0
addrs = pykd.dbgCommand("!dumpdomain").splitlines()
for x in addrs:
if x[:8] == "Assembly":
assemblies = assemblies + 1
print "### retrieving assembly " + x[-8:]
print pykd.dbgCommand("!dumpassembly " + x[-8:])
print "### found " + str(assemblies) + " assemblies."
dump_assemblies()
You will now be able to do a text search (CTRL + F) for the dll. Once you found it, you can find the offset next to the module name (looks something like 12327C8 C:\Program Files\MyApp\MyDLL.dll).
You can now dump the module using:
!dumpmodule 12327C8
which will lead you to:
Name: C:\Program Files\MyApp\MyDLL.dll
Attributes: PEFile
Assembly: 131a22e2
LoaderHeap: 00000000
TypeDefToMethodTableMap: 19220010
TypeRefToMethodTableMap: 134303e0
MethodDefToDescMap: 13430740
FieldDefToDescMap: 13433964
MemberRefToDescMap: 134350c8
FileReferencesMap: 13435918
AssemblyReferencesMap: 1343591c
MetaData start address: 1c1aaa4c (4248 bytes)
Now you can check the metadata of the dll simply with:
dc 1c1aaa4c 1c1aaa4c + 4248
That's the closest I could get to more information about the DLL. Unfortunately the file version was not written there, just some other more generic info. Anyway, I will try to get in touch with the customer again. Thanks for the answers anyway.
I have a problem with symbol files. I experimented with the symbol file path and set the path as follows:
srv*c:\symbols*http://msdl.microsoft.com/download/symbols;C:\Users\myuser\Desktop\driver2\objchk_win7_x86\i386
But afterwards I changed it to the following:
srv*c:\symbols*http://msdl.microsoft.com/download/symbols;C:\Users\myuser\Desktop\mydriver\objchk_win7_x86\i386
I changed the driver2 with mydriver in the path: this is the path where the .pdb file for my driver is located. The problem is that .sympath prints the right path as shown below:
kd> .sympath
Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols;C:\Users\myuser\Desktop\mydriver\objchk_win7_x86\i386
Expanded Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols;c:\users\myuser\desktop\mydriver\objchk_win7_x86\i386
But the symbols for the driver are still not found. If I run .reload command, we can see that WinDbg is looking for .pdb in driver2/ directory instead of mydriver/ directory.
kd> .reload /f mydriver.sys
SYMSRV: c:\symbols\mydriver.pdb\3D655E533B0449A38D7AB0AF637CE9201\mydriver.pdb not found
SYMSRV: http://msdl.microsoft.com/download/symbols/mydriver.pdb/3D655E533B0449A38D7AB0AF637CE9201/mydriver.pdb not found
SYMSRV: c:\users\myuser\desktop\mydriver\objchk_win7_x86\i386\mydriver.pdb\3D655E533B0449A38D7AB0AF637CE9201\mydriver.pdb not found
DBGHELP: c:\users\myuser\desktop\driver2\objchk_win7_x86\i386\mydriver.pdb - file not found
*** ERROR: Module load completed but symbols could not be loaded for mydriver.sys
DBGHELP: mydriver - no symbols loaded
I've deleted all the workspaces, closed WinDbg, restarted Windows, but the driver2/ entry is still there: it must be in the default workspace's cache or somewhere. How can I delete the whole personal settings of WinDbg including those caches, so I can restart WinDbg and be gone with the driver2/ path and make it use mydriver/.
I could also solve the problem with renaming the mydriver/ directory back into driver2/, but I don't want to solve the problem like that. I want to understand what's going on and solve it the best I can.
!sym noisy
will tell you why it did not want to load the pdb. Perhaps you did rebuild your driver and the pdb guid or pdb age does no longer match.
If you are sure that you have built the same source files you can force loading of your pdb by
.reload /i /f yourdriver.sys
/i is the magic switch to load also mismatched pdbs. This switch will not load any driver from your symbol server but it will consider only local file paths to load your driver. Also symbol store directories (SRV*) are not considered since there would be many versions to choose from. But if your .sympath directly points to your pdb it will be loaded.
Is the driver originally compiled & built in the driver2 path? What is the location of mydriver.sys?
For example assume I have symbol path 'c:\users\rahulsundar\desktop' set and try to load ntdll.dll, then it displays below error,
0:000> .reload ntdll.dll
DBGHELP: c:\users\rahulsundar\desktop\ntdll.pdb - file not found
DBGHELP: c:\users\rahulsundar\desktop\dll\ntdll.pdb - file not found
DBGHELP: c:\users\rahulsundar\desktop\symbols\dll\ntdll.pdb - file not found
DBGHELP: C:\Windows\SYSTEM32\ntdll.pdb - file not found
DBGHELP: ntdll.pdb - file not found
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll -
DBGHELP: ntdll - export symbols
Note: Windbg by default searches ntdll.pdb from the same location 'C:\Windows\SYSTEM32'
One way to solve the current issue, from the log its clear that windbg expects pdb file in directory 'c:\users\myuser\desktop\mydriver\objchk_win7_x86\i386\mydriver.pdb\3D655E533B0449A38D7AB0AF637CE9201\mydriver.pdb'.
So manually create directory till
'mydriver.pdb\3D655E533B0449A38D7AB0AF637CE9201' and place pdb file
there.
This is just a standard way( binaryfoldername\hashid\pdbfile ) that Windows expects a symbol for a binary.
Better way to solve this - turn on sim noisy and look at path
at .reload /f my_driver.sys
or add new path to .sympath[+] path/to/pdb and do the same