Debugging XP SP2/SP3 native application symbol path issue - debugging

Question

When I tried to validate, if WinDbg has been setup properly, I opened the executable C:\WINDOWS\NOTEPAD.exe and tried to check the loaded modules.
First of all, these are the environment variables set:
_NT_SYMBOL_PATH: c:\mysymbols;SRV*c:\symbols*https://msdl.microsoft.com/download/symbols
_NT_EXECUTABLE_IMAGE_PATH: SRV*c:\symbols
This is the WinDbg output:
CommandLine: C:\WINDOWS\NOTEPAD.EXE
Symbol search path is: SRV*C:\symbols*https://msdl.microsoft.com/download/symbols;c:\mysymbols
Executable search path is: SRV*c:\symbols
ModLoad: 01000000 01014000 notepad.exe
ModLoad: 7c910000 7c9c9000 ntdll.dll
ModLoad: 7c800000 7c908000 C:\WINDOWS\system32\kernel32.dll
ModLoad: 76350000 7639a000 C:\WINDOWS\system32\comdlg32.dll
ModLoad: 77da0000 77e4a000 C:\WINDOWS\system32\ADVAPI32.dll
ModLoad: 77e50000 77ee3000 C:\WINDOWS\system32\RPCRT4.dll
ModLoad: 77fc0000 77fd1000 C:\WINDOWS\system32\Secur32.dll
ModLoad: 773a0000 774a3000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.6028_x-ww_61e65202\COMCTL32.dll
ModLoad: 77be0000 77c38000 C:\WINDOWS\system32\msvcrt.dll
ModLoad: 77ef0000 77f39000 C:\WINDOWS\system32\GDI32.dll
ModLoad: 7e360000 7e3f1000 C:\WINDOWS\system32\USER32.dll
ModLoad: 77f40000 77fb6000 C:\WINDOWS\system32\SHLWAPI.dll
ModLoad: 7e670000 7ee91000 C:\WINDOWS\system32\SHELL32.dll
ModLoad: 72f70000 72f96000 C:\WINDOWS\system32\WINSPOOL.DRV
(ef8.f6c): Break instruction exception - code 80000003 (first chance)
eax=001a1eb4 ebx=7ffd5000 ecx=00000007 edx=00000080 esi=001a1f48 edi=001a1eb4
eip=7c91120e esp=0007fb20 ebp=0007fc94 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll -
ntdll!DbgBreakPoint:
7c91120e cc int 3
I switched on sym noisy mode, called .reload and got the continued output when I tried to list the loaded modules:
0:000> !sym noisy
noisy mode - symbol prompts on
0:000> .reload
Reloading current modules
..............
DBGHELP: c:\mysymbols\ntdll.pdb - mismatched pdb
DBGHELP: c:\mysymbols\symbols\dll\ntdll.pdb - file not found
DBGHELP: c:\mysymbols\dll\ntdll.pdb - file not found
SYMSRV: Die Serververbindung wurde zurückgesetzt.
SYMSRV: c:\symbols\ntdll.pdb\A618C674A4FC40F5B1781029C2C7F68E2\ntdll.pdb not found
SYMSRV: https://msdl.microsoft.com/download/symbols/ntdll.pdb/A618C674A4FC40F5B1781029C2C7F68E2/ntdll.pdb not found
DBGHELP: C:\WINDOWS\system32\ntdll.pdb - file not found
DBGHELP: ntdll.pdb - file not found
DBGHELP: Couldn't load mismatched pdb for C:\WINDOWS\system32\ntdll.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll -
DBGHELP: ntdll - export symbols
0:000> lm
start end module name
01000000 01014000 notepad (deferred)
72f70000 72f96000 WINSPOOL (deferred)
76350000 7639a000 comdlg32 (deferred)
773a0000 774a3000 COMCTL32 (deferred)
77be0000 77c38000 msvcrt (deferred)
77da0000 77e4a000 ADVAPI32 (deferred)
77e50000 77ee3000 RPCRT4 (deferred)
77ef0000 77f39000 GDI32 (deferred)
77f40000 77fb6000 SHLWAPI (deferred)
77fc0000 77fd1000 Secur32 (deferred)
7c800000 7c908000 kernel32 (deferred)
7c910000 7c9c9000 ntdll (export symbols) C:\WINDOWS\system32\ntdll.dll
7e360000 7e3f1000 USER32 (deferred)
7e670000 7ee91000 SHELL32 (deferred)
The line 7c910000 7c9c9000 ntdll (export symbols) C:\WINDOWS\system32\ntdll.dll shows, that the symbol files couldn't get loaded successfully.
I have downloaded and installed the symbols (like WindowsXP-KB835935-SP2-slp-Symbols) but still the PDB files don't seem to match with my ntdll.dll build. But why is that the case? Is there any chance I can get this to work?

Answer 1

As I couldn't stop pondering about the root of the problem, I reread the output above:
SYMSRV: c:\symbols\ntdll.pdb\A618C674A4FC40F5B1781029C2C7F68E2\ntdll.pdb not found
SYMSRV: https://msdl.microsoft.com/download/symbols/ntdll.pdb/A618C674A4FC40F5B1781029C2C7F68E2/ntdll.pdb not found
The debugger is looking for a symbol file having the GUID A618C674A4FC40F5B1781029C2C7F68E2, it even outputs an URI it is trying to load the pdb from. Thus, I tried to download the pdb manually, which worked, created the GUID directory manually, renamed the blob file to ntdll.pdb and placed it within the GUID-directory.
As a result, it did finally work, as the following output shows:
:000> .reload
Reloading current modules
............
DBGHELP: c:\mysymbols\ntdll.pdb - mismatched pdb
DBGHELP: c:\mysymbols\symbols\dll\ntdll.pdb - file not found
DBGHELP: c:\mysymbols\dll\ntdll.pdb - file not found
DBGHELP: ntdll - public symbols
c:\symbols\ntdll.pdb\A618C674A4FC40F5B1781029C2C7F68E2\ntdll.pdb
..
0:000> lm
start end module name
01000000 01014000 notepad (deferred)
72f70000 72f96000 WINSPOOL (deferred)
76350000 7639a000 comdlg32 (deferred)
773a0000 774a3000 COMCTL32 (deferred)
77be0000 77c38000 msvcrt (deferred)
77da0000 77e4a000 ADVAPI32 (deferred)
77e50000 77ee2000 RPCRT4 (deferred)
77ef0000 77f38000 GDI32 (deferred)
77f40000 77fb6000 SHLWAPI (deferred)
77fc0000 77fd1000 Secur32 (deferred)
7c800000 7c907000 kernel32 (deferred)
7c910000 7c9c9000 ntdll (pdb symbols) c:\symbols\ntdll.pdb\A618C674A4FC40F5B1781029C2C7F68E2\ntdll.pdb
7e360000 7e3f0000 USER32 (deferred)
7e670000 7ee90000 SHELL32 (deferred)
EDIT: I was finally able to get the automatic download from the symbol server working! I always thought about it in the beginning, but did not believe it could be the reason: With a newer version 6.12.0002.633 it does work as expected, while with the 6.6.07.5 it did not.