I got a dump file of my Application Process.
And I processed it with debugdiag 2.0 and windbg.
But, I got a different thread call stack.
So, I am very confused.
Can anybody tell me why this happen?
below is windbg's thread call stack
. 0 Id: 16ec.1760 Suspend: 0 Teb: 00000000`7ec1a000 Unfrozen
# Call Site
00 wow64cpu!CpupSyscallStub
01 wow64cpu!ReadWriteFileFault
02 wow64!RunCpuSimulation
03 wow64!Wow64LdrpInitialize
04 ntdll!LdrpInitializeProcess
05 ntdll!_LdrpInitialize
06 ntdll!LdrInitializeThunk
And below is debugdiag's thread call stack
Thread 0 - System ID 5984
Entry point IMCM+1e05f
Create time 2018-04-24 AM 10:54:21
Time spent in user mode 0 Days 00:00:00.125
Time spent in kernel mode 0 Days 00:00:00.046
This thread is not fully resolved and may or may not be a problem. Further analysis of these threads may be required.
ntdll_77720000!NtReadFile+c
KERNELBASE!ReadFile+79
msvcr110!_read_nolock+272
msvcr110!_read+a9
msvcr110!_filbuf+70
msvcr110!_fgetwc_nolock+114
msvcr110!_getws_helper+63
msvcr110!_getws_s+10
IMCM+18ec
IMSvcSrvLib!CSvcApp::Run+5c
IMComLib!CIMBaseApp::Main+62
IMCM+15515
IMCM+1dff7
kernel32!BaseThreadInitThunk+24
ntdll_77720000!__RtlUserThreadStart+2f
ntdll_77720000!_RtlUserThreadStart+1b
I was trying to build MIT Kerberos In Windows 2007 (Windows Server Enterprise) Service Pack 2 32 bit system. After adding a few flags specific to posix errors I was able to build it in Windows 7 (along with working kinit and klist programs). However in win 2007 all exes generated crash whenever I attempt to execute them. I had used Microsoft visual studio 2008 with Microsoft SDK v6 for both builds.
Crash code in event viewer: Exception code: 0xc000041d and occasionally 0xc00008c
Fault offset: 0x76e011f1
After enabling all possible checks in gflags and running kinit, I noticed a message saying unable to start application due to incorrect security permissions. I changed compatibility mode to xp3 and ran as administrator but no luck.
I then used sxstrace to determine any link time inconsistencies. I didnt find even a single line in my parsed trace file. I then used dependency walker and it wasnt able to find any errors.
I then used procdump and windbg to get the dump of the problem. Unfortunately I havent been able to locate a suitable pdb for nt.dll. This is what i found after attempting to unwind the core dump stack (kp command):-
0018975c 64754d57 user32!GetProcessWindowStation+0x15
0018a8c0 64755d08 msvcr90d!CrtDbgReport+0x437
0018f954 64754992 msvcr90d!VCrtDbgReportA+0x7d8
0018f974 6475494b msvcr90d!CrtDbgReport+0x72
0018f99c 646bc34d msvcr90d!CrtDbgReport+0x2b
0018f9d0 646bc812 msvcr90d!get_pgmptr+0x1bd
0018fa08 646bc711 msvcr90d!_getmainargs+0x182
0018fa1c 76fc99a0 msvcr90d!_getmainargs+0x81
0018fa3c 76fcd939 ntdll!RtlQueryEnvironmentVariable+0x241
0018fb30 76fd686c ntdll!LdrResSearchResource+0xb4d
0018fcb0 76fd5326 ntdll!RtlGetNtVersionNumbers+0x9b
0018fd00 76fc9ef9 ntdll!RtlSetUnhandledExceptionFilter+0x50
0018fd10 00000000 ntdll!LdrInitializeThunk+0x10
I dont quite understand what this means and I have no idea what on earth is going on. I dont have too much proficiency in using windbg
Is there anything else that anyone can suggest me to narrow down the root cause of the issue? Even after I copy the 2k7 built binaries to my local win 7 machine and it still crashes with the same stack.
Edit: after running .symfix, .reload and then analyze -v I got the following output in windbg console:-
*** WARNING: Unable to verify checksum for klist.exe
*** ERROR: Module load completed but symbols could not be loaded for klist.exe
FAULTING_IP:
+0
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
FAULTING_THREAD: 000014bc
PROCESS_NAME: klist.exe
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
APP: klist.exe
BUGCHECK_STR: APPLICATION_FAULT_STATUS_BREAKPOINT_AFTER_CALL
PRIMARY_PROBLEM_CLASS: STATUS_BREAKPOINT_AFTER_CALL
DEFAULT_BUCKET_ID: STATUS_BREAKPOINT_AFTER_CALL
LAST_CONTROL_TRANSFER: from 6475450f to 74c49eff
STACK_TEXT:
00189718 6475450f 0018973c 0018a8c0 64754cc0 user32!NtUserGetProcessWindowStation+0x15
0018975c 64754d57 001898b0 64696070 00012012 msvcr90d!__crtMessageBoxA+0x14f
0018a8c0 64755d08 00000001 00000000 00000000 msvcr90d!__crtMessageWindowA+0x3b7
0018f954 64754992 00000001 00000000 00000000 msvcr90d!_VCrtDbgReportA+0x7d8
0018f974 6475494b 00000001 00000000 00000000 msvcr90d!_CrtDbgReportV+0x22
0018f99c 646bc34d 00000001 00000000 00000000 msvcr90d!_CrtDbgReport+0x2b
0018f9d0 646bc812 00000022 6e76fe50 0018faec msvcr90d!_NMSG_WRITE+0x6d
0018fa08 646bc711 64680000 00000001 0018fd24 msvcr90d!__CRTDLL_INIT+0xf2
0018fa1c 76fc99a0 64680000 00000001 0018fd24 msvcr90d!_CRTDLL_INIT+0x21
0018fa3c 76fcd939 646bc6f0 64680000 00000001 ntdll!LdrpCallInitRoutine+0x14
0018fb30 76fd686c 0018fd24 7efdd000 7efde000 ntdll!LdrpRunInitializeRoutines+0x26f
0018fcb0 76fd5326 0018fd24 76f90000 734dc02c ntdll!LdrpInitializeProcess+0x1400
0018fd00 76fc9ef9 0018fd24 76f90000 00000000 ntdll!_LdrpInitialize+0x78
0018fd10 00000000 0018fd24 76f90000 00000000 ntdll!LdrInitializeThunk+0x10
FOLLOWUP_IP:
msvcr90d!__crtMessageBoxA+14f [f:\dd\vctools\crt_bld\self_x86\crt\src\crtmbox.c # 121]
6475450f 8945ec mov dword ptr [ebp-14h],eax
FAULTING_SOURCE_LINE: f:\dd\vctools\crt_bld\self_x86\crt\src\crtmbox.c
FAULTING_SOURCE_FILE: f:\dd\vctools\crt_bld\self_x86\crt\src\crtmbox.c
FAULTING_SOURCE_LINE_NUMBER: 121
SYMBOL_STACK_INDEX: 1
SYMBOL_NAME: msvcr90d!__crtMessageBoxA+14f
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: msvcr90d
IMAGE_NAME: msvcr90d.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 488ef6c7
STACK_COMMAND: dt ntdll!LdrpLastDllInitializer BaseDllName ; dt ntdll!LdrpFailureData ; ~0s; .ecxr ; kb
FAILURE_BUCKET_ID: STATUS_BREAKPOINT_AFTER_CALL_80000003_msvcr90d.dll!__crtMessageBoxA
BUCKET_ID: APPLICATION_FAULT_STATUS_BREAKPOINT_AFTER_CALL_msvcr90d!__crtMessageBoxA+14f
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/klist_exe/4_0_0_0/533e75fb/unknown/0_0_0_0/bbbbbbb4/80000003/00000000.htm?Retriage=1
Followup: MachineOwner
Edit: After running in Visual Studio I got the following output:-
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\klist.exe', Symbols loaded.
'klist.exe': Loaded 'C:\Windows\SysWOW64\ntdll.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\kernel32.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\KernelBase.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\sysfer.dll'
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\k5sprt32.dll', Symbols loaded.
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\msvcr90d.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\ws2_32.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\msvcrt.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\rpcrt4.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\sspicli.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\cryptbase.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\sechost.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\nsi.dll'
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\krb5_32.dll', Symbols loaded.
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\comerr32.dll', Symbols loaded.
'klist.exe': Loaded 'C:\Windows\SysWOW64\user32.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\gdi32.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\lpk.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\usp10.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\advapi32.dll'
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\wshelp32.dll', Symbols loaded.
'klist.exe': Loaded 'C:\Windows\SysWOW64\dnsapi.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\shell32.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\shlwapi.dll'
First-chance exception at 0x74c49eff in klist.exe: 0xC0000005: Access violation reading location 0x00000250.
*** An Access Violation occurred in "C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\klist.exe" :
The instruction at 0000000076E011F1 tried to read from an invalid address, 0000000000000250
*** enter .exr 000000000008E970 for the exception record
*** enter .cxr 000000000008E480 for the context
*** then kb to get the faulting stack
Unhandled exception at 0x74c49eff in klist.exe: 0xC000041D: An unhandled exception was encountered during a user callback.
> kb
Index Function
--------------------------------------------------------------------------------
*1 user32.dll!74c49eff()
2 [Frames below may be incorrect and/or missing, no symbols loaded for user32.dll]
3 user32.dll!74c49eff()
4 msvcr90d.dll!58f8450f()
5 msvcr90d.dll!58f84d57()
I cant get klist or krb5 dlls in the stack at all. Since klist or any other mit kerb dll does not appear in this section, I am unable to load check their symbols. This is very frustrating, I will attempt to build my own sample program and check for issues. Btw did I miss any analysis steps?
Edit : After checking for first argument to crtmessagebox I got :-
001898b0 "Debug Error!..Program: C:\WS\TPL"
001898d0 "\src\MitKerberos\1.11.1\BUILDDEB"
001898f0 "UG\bin\klist.exe..R6034..An appl"
00189910 "ication has made an attempt to l"
00189930 "oad the C runtime library withou"
00189950 "t using a manifest..This is an u"
00189970 "nsupported way to load Visual C+"
00189990 "+ DLLs. You need to modify your "
001899b0 "application to build with a mani"
001899d0 "fest..For more information, see "
001899f0 "the "Visual C++ Libraries as Sha"
00189a10 "red Side-by-Side Assemblies" top"
As far as I understand the program responsible for this is mt.exe and I had run it.
I am unable to figure this problem out. Symbol is not being resolved
Deployment
There are number of exes of my system deployed on a network path. All users run those exes from that shared network path. This was working fine two weeks ago but now some of those exes have started crashing. There is no fix pattern of being crashed, it happens to any user, anytime during any activity.
Troubleshooting
I have got the dump of one of them, i tried WinDbg and got following
Microsoft (R) Windows Debugger Version 6.2.9200.16384 X86
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\crash\RNS1000.exe.mdmp]
User Mini Dump File: Only registers, stack and portions of memory are available
Symbol search path is: SRV*c:\crash*http://msdl.microsoft.com/download/symbols;c:\crash
Executable search path is:
Windows XP Version 2600 (Service Pack 3) MP (2 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
Machine Name:
Debug session time: Wed Oct 10 15:36:36.000 2012 (UTC + 5:00)
System Uptime: not available
Process Uptime: 0 days 7:12:54.000
................................................................
.........................................................
Loading unloaded module list
.......
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(924.85c): In-page I/O error ffffffffc000020c - code c0000006 (first/second chance not available)
eax=02060000 ebx=7c90fe01 ecx=00001000 edx=7c90e4f4 esi=000003a0 edi=00000000
eip=7c90e4f4 esp=0013afdc ebp=0013b040 iopl=0 nv up ei ng nz ac pe cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00200297
ntdll!KiFastSystemCallRet:
7c90e4f4 c3 ret
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
GetUrlPageData2 (WinHttp) failed: 12007.
FAULTING_IP:
RNS1000+55f610
0095f610 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0095f610 (RNS1000+0x0055f610)
ExceptionCode: c0000006 (In-page I/O error)
ExceptionFlags: 00000000
NumberParameters: 3
Parameter[0]: 00000008
Parameter[1]: 0095f610
Parameter[2]: c000020c
Inpage operation failed at 0095f610, due to I/O error c000020c
DEFAULT_BUCKET_ID: SOFTWARE_NX_FAULT
PROCESS_NAME: RNS1000.exe
ERROR_CODE: (NTSTATUS) 0xc0000006 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The required data was not placed into memory because of an I/O error status of "0x%08lx".
EXCEPTION_CODE: (NTSTATUS) 0xc0000006 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The required data was not placed into memory because of an I/O error status of "0x%08lx".
EXCEPTION_PARAMETER1: 00000008
EXCEPTION_PARAMETER2: 0095f610
EXCEPTION_PARAMETER3: c000020c
IO_ERROR: (NTSTATUS) 0xc000020c - The transport connection is now disconnected.
ADDITIONAL_DEBUG_TEXT: Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]
LAST_CONTROL_TRANSFER: from 00000000 to 00000000
APP: rns1000.exe
FAULTING_THREAD: ffffffff
PRIMARY_PROBLEM_CLASS: SOFTWARE_NX_FAULT
BUGCHECK_STR: APPLICATION_FAULT_SOFTWARE_NX_FAULT
STACK_TEXT:
00000000 00000000 hardware_disk!Unknown+0x0
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: hardware_disk!Unknown
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: hardware_disk
DEBUG_FLR_IMAGE_TIMESTAMP: 0
STACK_COMMAND: ** Pseudo Context ** ; kb
FAILURE_BUCKET_ID: SOFTWARE_NX_FAULT_c0000006_hardware_disk!Unknown
BUCKET_ID: APPLICATION_FAULT_SOFTWARE_NX_FAULT_hardware_disk!Unknown
IMAGE_NAME: hardware_disk
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/RNS1000_exe/2_0_0_5/4f17b9d2/RNS1000_exe/2_0_0_5/4f17b9d2/c0000006/0055f610.htm?Retriage=1
Followup: MachineOwner
---------
I am expecting RNS1000+55f610 to be resolved to one of my programs function but it has not been resolved. The sysmbol path contains exe, pdb and mdmp.
Please tell me why has it not been resolved? what wrong am i doing?
The key part here is the In-page I/O error. The underlying disk/network drive disappeared.
The crash occurs some time later when it tries to page back in part of the executable, but it no longer has a valid file handle/connection.
The only fix is to run it locally or make sure the disk doesn't disappear while they're running.
More generally, you can get VB to create the info files for native debugging using the "Create symbolic debug info" option in the project's Compile settings. This can only be done before the fact though and won't help with debugging an existing build.
I attached WinDbg to a running process and had the process crashed (I have a separate question re. that case). Once the program crashed, WinDbg stopped and allowed me to debug the program. I took a crash dump for further investigation with a command ".dump /ma".
The program was compiled as "Any CPU" and I used WinDbg x64 to take the dump. Now I open WinDbg x64 on the same computer again and open the crash dump. Here is what it says:
Loading Dump File [C:\crashdump.dmp]
User Mini Dump File with Full Memory: Only application data is available
Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: WinNt, suite: SingleUserTS
Machine Name:
Debug session time: Mon Aug 15 10:24:57.000 2011 (UTC + 1:00)
System Uptime: 17 days 0:54:39.021
Process Uptime: 12 days 14:01:31.000
................................................................
...............................................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1be0.b78): Access violation - code c0000005 (first/second chance not available)
*** WARNING: symbols timestamp is wrong 0x4dd2333e 0x4da4281c for clr.dll
clr!WKS::gc_heap::find_first_object+0x92:
000007fe`ea129a1d f70100000080 test dword ptr [rcx],80000000h ds:00000000`00003d80=????????
Then I try to load SOS by ".load sos clr" and it errors in:
The call to LoadLibrary(sos clr) failed, Win32 error 0n2
"The system cannot find the file specified."
Please check your debugger configuration and/or network access.
Trying with ".loadby sos clr" and it works. Now I want to see the stack with "!clrstack" and stick here:
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of clr.dll is
in the version directory
3) or, if you are debugging a dump file, verify that the file
mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file.
For example, an IA64 dump file must be debugged on an IA64
machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.
I tried ".symfix" and ".reload":
0:027> .symfix
0:027> .reload
..................*** WARNING: symbols timestamp is wrong 0x4dd2333e 0x4da4281c for clr.dll
..............................................
...............................................................
Stuck. At the same time when the process is running under WinDgb I can pause the execution, load SOS
and execute "!clrstack" command successfully.
Any ideas?
Thank you.
UPDATE - Followed the steps provided in the second answer, still doesn't work.
1) My Symbol Path: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols;srv*
2) CLR loaded: 4.0.30319.237:
0:027> lm v clr
Unknown option 'r'
start end module name
00000000`77b60000 00000000`77d09000 ntdll (pdb symbols) c:\symbols\ntdll.pdb\6192BFDB9F04442995FFCB0BE95172E12\ntdll.pdb
Loaded symbol image file: ntdll.dll
Image path: C:\Windows\System32\ntdll.dll
Image name: ntdll.dll
Timestamp: Sat Nov 20 13:11:21 2010 (4CE7C8F9)
CheckSum: 001B55EA
ImageSize: 001A9000
File version: 6.1.7601.17514
Product version: 6.1.7601.17514
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: ntdll.dll
OriginalFilename: ntdll.dll
ProductVersion: 6.1.7601.17514
FileVersion: 6.1.7601.17514 (win7sp1_rtm.101119-1850)
FileDescription: NT Layer DLL
LegalCopyright: © Microsoft Corporation. All rights reserved.
000007fe`e9fb0000 000007fe`ea915000 clr # (pdb symbols) c:\symbols\clr.pdb\1A7EA01DA29549DAB2B0BD012A6C5BA12\clr.pdb
Loaded symbol image file: clr.dll
Image path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Image name: clr.dll
Timestamp: Tue May 17 09:35:10 2011 (4DD2333E)
CheckSum: 00967144
ImageSize: 00965000
File version: 4.0.30319.237
Product version: 4.0.30319.237
File flags: 8 (Mask 3F) Private
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® .NET Framework
InternalName: clr.dll
OriginalFilename: clr.dll
ProductVersion: 4.0.30319.235
FileVersion: 4.0.30319.235 (RTMGDR.030319-2300)
PrivateBuild: DDBLD240
FileDescription: Microsoft .NET Runtime Common Language Runtime - WorkStation
LegalCopyright: © Microsoft Corporation. All rights reserved.
Comments: Flavor=Retail
3) "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscordacwks.dll" has version 4.0.30319.239
4) I found that when I load the dump into WinDbg it loads the correct "mscordacwks.dll" from the web, thus in the folder "C:\symbols\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD2333E965000" I have the file "mscordacwks_AMD64_AMD64_4.0.30319.237.dll".
5)
0:027> .cordll -ve -u -l
CLR DLL status: No load attempts
6)
0:027> !sym noisy
noisy mode - symbol prompts on
0:027> .restart
Loading Dump File [C:\crashdump.dmp]
User Mini Dump File with Full Memory: Only application data is available
DBGHELP: Symbol Search Path: srv*;srv*c:\symbols*http://msdl.microsoft.com/download/symbols
DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols
DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: srv*;SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: WinNt, suite: SingleUserTS
Machine Name:
Debug session time: Mon Aug 15 10:24:57.000 2011 (UTC + 1:00)
System Uptime: 17 days 0:54:39.021
Process Uptime: 12 days 14:01:31.000
................................................................
...............................................................
DBGHELP: ntdll - public symbols
C:\Program Files\Debugging Tools for Windows (x64)\sym\ntdll.pdb\6192BFDB9F04442995FFCB0BE95172E12\ntdll.pdb
DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols
DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1be0.b78): Access violation - code c0000005 (first/second chance not available)
*** WARNING: symbols timestamp is wrong 0x4dd2333e 0x4da4281c for clr.dll
DBGHELP: clr - public symbols
C:\Program Files\Debugging Tools for Windows (x64)\sym\clr.pdb\1A7EA01DA29549DAB2B0BD012A6C5BA12\clr.pdb
clr!WKS::gc_heap::find_first_object+0x92:
000007fe`ea129a1d f70100000080 test dword ptr [rcx],80000000h ds:00000000`00003d80=????????
7)
0:027> !clrstack
SYMSRV: C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD2333E965000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll not found
SYMSRV: mscordacwks_AMD64_AMD64_4.0.30319.237.dll from http://msdl.microsoft.com/download/symbols: 502892 bytes - copied
DBGHELP: C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD2333E965000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll cached to C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD233F317b000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll
DBGHELP: C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD233F317b000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll - OK
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of clr.dll is
in the version directory
3) or, if you are debugging a dump file, verify that the file
mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file.
For example, an IA64 dump file must be debugged on an IA64
machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.
I hit this regularly when debugging with minidumps from the site. Quite how it's happened in your case I'm not sure. Usually, it happens when the version of CLR that was loaded when the dump was taken is not available on your debugging machine. In your case, they're the same machine, so it should all probably just work. I'm sure there will be others who can explain exactly why it isn't.
In the meantime, here's what I do with my site dumps. Windbg is looking for "the right version" of mscordacwks.dll. So we give it that version and tell it where to look for it.
First - if I spoof all of this, by deleting mscordacwks.dll, windbg goes off and loads it from the Microsoft symbol server, so do make sure your symbols are set up correctly to download symbols from the Microsoft symbol server and give it another go.
Now - assuming that didn't work, check exactly which version is the "right version". List the module info with "lm v clr" and check your CLR version that is ACTUALLY loaded. Mine is 4.0.30319.239. Ok - now find that version of mscordacwks.dll. Let's assume it can be found in the normal .NET framework installation on your machine (C:\Windows\Microsoft.NET\Framework64\v4.0.30319). Do check the version matches exactly (right-click, properties etc)! Take it and put it in a safe place (I use D:\Symbols\_Images). Follow the instructions that windbg gave you on renaming the file. mscordacwks_.dll would be mscordacwks_AMD64_AMD64_4.0.30319.239.dll.
Now set up your executable image path (".exepath D:\Symbols\_Images") so windbg knows where you've put it.
You've now got "the right version of mscordacwks", and renamed it so that Windbg knows what it's looking for, and told it where you've put it.
If that STILL isn't working, then try ".cordll -ve -u -l" and also "!sym noisy" to turn on verbose logging of both the cordll load and the symbol server, then try !CLRStack again. Maybe the output of those two commands will tell you exactly what it's trying to load and you can figure out why it won't do it...
I just spent the day debugging a bunch of cases where we ran into this scenario. The SOS+CLR on the same box as the crash were unable to load within WinDbg, and "lm v" reported two different versions for the same module:
0:011> lm vM *clr.dll
start end module name
000007fe`f2f50000 000007fe`f38b0000 clr # (pdb symbols) c:\symbols\clr.pdb\EDFF900AC9B94C1D9B32696A7759891A2\clr.pdb
Loaded symbol image file: clr.dll
Image path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Image name: clr.dll
Timestamp: Sun Apr 21 03:36:04 2013 (5173C114)
CheckSum: 0095F379
ImageSize: 00960000
File version: 4.0.30319.18052
Product version: 4.0.30319.18052
File flags: 8 (Mask 3F) Private
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® .NET Framework
InternalName: clr.dll
OriginalFilename: clr.dll
ProductVersion: 4.0.30319.18047
FileVersion: 4.0.30319.18047 built by: FX45RTMGDR
PrivateBuild: DDBLD320
FileDescription: Microsoft .NET Runtime Common Language Runtime - WorkStation
LegalCopyright: © Microsoft Corporation. All rights reserved.
Comments: Flavor=Retail
Backing Details
The file Timestamp (0x5173C114), Checksum (0x0095F379), and Version (4.0.30319.18052) stored in the MINIDUMP_MODULE structure in the minidump's module-list-stream was for the newer CLR. Cracking open the minidump file myself and looking directly at the stream data:
MINIDUMP_MODULE : (pack:8 size:112)
+0x000 .BaseOfImage UInt64 : 8791579230208 (0x7FEF2F50000)
+0x008 .SizeOfImage UInt32 : 9830400 (0x960000)
+0x00C .CheckSum UInt32 : 9827193 (0x95F379)
+0x010 .TimeDateStamp UInt32 : 1366540564 (0x5173C114)
+0x014 .ModuleNameRva UInt32 : 107828 (0x1A534)
+0x018 .VersionInfo tagVS_FIXEDFILEINFO : (pack:8 size:52)
+0x000 .dwSignature UInt32 : 4277077181 (0xFEEF04BD)
+0x004 .dwStrucVersion UInt32 : 65536 (0x10000)
+0x008 .dwFileVersionMS UInt32 : 262144 (0x40000)
+0x00C .dwFileVersionLS UInt32 : 1987004036 (0x766F4684)
Splitting the high and low words out of dwFileVersionMS we get 4 and 0.
Splitting the high and low words out of dwFileVersionLS we get 30319 and 18052.
Using dumpchk.exe, and looking at the module details in the PEB, we can see a different Timestamp (0x515530CE), one that actually corresponds to the older (18047) version:
7fef2f50000 515530ce Mar 28 23:12:30 2013 C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Looking at the raw memory in the crash dump where clr.dll is loaded, you can see the Checksum (0x00965F80) and Timestamp (0x515530CE) of version 4.0.30319.18047:
0:011> db 000007fe`f2f50000
000007fe`f2f50000 4d 5a 90 00 03 00 00 00-04 00 00 00 ff ff 00 00 MZ..............
000007fe`f2f50010 b8 00 00 00 00 00 00 00-40 00 00 00 00 00 00 00 ........#.......
000007fe`f2f50020 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
000007fe`f2f50030 00 00 00 00 00 00 00 00-00 00 00 00 18 01 00 00 ................
000007fe`f2f50040 0e 1f ba 0e 00 b4 09 cd-21 b8 01 4c cd 21 54 68 ........!..L.!Th
000007fe`f2f50050 69 73 20 70 72 6f 67 72-61 6d 20 63 61 6e 6e 6f is program canno
000007fe`f2f50060 74 20 62 65 20 72 75 6e-20 69 6e 20 44 4f 53 20 t be run in DOS
000007fe`f2f50070 6d 6f 64 65 2e 0d 0d 0a-24 00 00 00 00 00 00 00 mode....$.......
0:011> db
000007fe`f2f50080 39 e4 28 ed 7d 85 46 be-7d 85 46 be 7d 85 46 be 9.(.}.F.}.F.}.F.
000007fe`f2f50090 81 f2 f8 be 79 85 46 be-81 f2 fa be 74 85 46 be ....y.F.....t.F.
000007fe`f2f500a0 74 fd c5 be 73 85 46 be-74 fd c2 be c9 85 46 be t...s.F.t.....F.
000007fe`f2f500b0 ee 41 8d be 7f 85 46 be-e3 25 81 be 7c 85 46 be .A....F..%..|.F.
000007fe`f2f500c0 ee 41 88 be 6b 85 46 be-ee 41 89 be 78 85 46 be .A..k.F..A..x.F.
000007fe`f2f500d0 ee 41 8b be 64 85 46 be-7d 85 47 be ca 87 46 be .A..d.F.}.G...F.
000007fe`f2f500e0 81 f2 ff be 76 85 46 be-ee 41 9e be 70 87 46 be ....v.F..A..p.F.
000007fe`f2f500f0 ee 41 8c be 7c 85 46 be-ee 41 8f be 7c 85 46 be .A..|.F..A..|.F.
0:011>
000007fe`f2f50100 ee 41 8a be 7c 85 46 be-52 69 63 68 7d 85 46 be .A..|.F.Rich}.F.
000007fe`f2f50110 00 00 00 00 00 00 00 00-50 45 00 00 64 86 06 00 ........PE..d...
000007fe`f2f50120 ce 30 55 51 00 00 00 00-00 00 00 00 f0 00 22 20 .0UQ.........."
000007fe`f2f50130 0b 02 0b 00 00 90 69 00-00 c2 2b 00 00 00 00 00 ......i...+.....
000007fe`f2f50140 40 51 13 00 00 10 00 00-00 00 f5 f2 fe 07 00 00 #Q..............
000007fe`f2f50150 00 10 00 00 00 02 00 00-06 00 00 00 0a 00 00 00 ................
000007fe`f2f50160 06 00 00 00 00 00 00 00-00 e0 95 00 00 04 00 00 ................
000007fe`f2f50170 80 5f 96 00 02 00 60 01-00 00 10 00 00 00 00 00 ._....`.........
I also jumped ahead in memory and looked at the Version resource in memory and saw the 18047 version string.
So now we have a minidump with conflicting information about what version of clr.dll was actually in use.
What Caused It
I also found out that our IT department recently pushed out a handful of Windows Updates, so:
While an application was running, an update to the CLR was installed.
The file in C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ was updated to the newer version (4.0.30319.18052)
The application Crashed
MiniDumpWriteDump apparently used the file-on-disk information when storing the module list in the crash dump file (4.0.30319.18052)
WinDbg wasn't able to correlate the version stamped in the crash dump with what was in the process' memory as it had conflicting information.
To verify this, I manually modified the MINIDUMP_MODULE entry for clr.dll to change the Checksum, Timestamp, and Version from 18052 to 18047. After reloading the hacked up .dmp file in WinDbg and setting the exepath to the proper sos+clr dlls, I was able to successfully execute the sos commands and get a valid stack trace.
Bottom Line
We essentially ended up with a minidump file that has conflicting information about which version of clr.dll is loaded into the process, preventing SOS from identifying the correct clr engine.
This is likely a bug in MiniDumpWriteDump as it saves out the crash dump file. But should only happen when the backing version of the CLR has been updated while your application was running.
Sounds like you did a custom install of windbg and did not select all the extensions that you require. The Win32 error n2 is generally a sign of this problem.
Is the process which you are trying to debug a 32-bit? What does the task manager-processes listing say ? If its 32-bit process then you need to use 32-bit windbg.
Otherwise, I don't see why .load sos clr should fail.
ps: (windbg noob alert), so i apologize if this sounds lame. Just trying to help.
I'm a complete WinDbg newbie and I've been trying to debug a WindowsXP problem that a customer has sent me where our software and some third party software prevent windows from logging off. I've reproduced the problem and have verified that only when our software and the customers software are both installed (although not necessarily running at logoff) does the log off problem occur. I've observed that WM_ENDSESSION messages are not reaching the running windows when the user tries to log off and I know that the third party software uses a kernel driver.
I've been looking at the processes in WinDbg and I know that csrss.exe would normally send all the windows a WM_ENDSESSION message. When I ran:
!process 82356020 6
To look at csrss.exe's stack I can see:
WARNING: Frame IP not in any known module. Following frames may be wrong.
00000000 00000000 00000000 00000000 00000000 0x7c90e514
THREAD 8246d998 Cid 0248.02a0 Teb: 7ffd7000 Win32Thread: e1627008 WAIT: (WrUserRequest) UserMode Non-Alertable
8243d9f0 SynchronizationEvent
81fe0390 SynchronizationEvent
Not impersonating
DeviceMap e1004450
Owning Process 82356020 Image: csrss.exe
Attached Process N/A Image: N/A
Wait Start TickCount 1813 Ticks: 20748 (0:00:05:24.187)
Context Switch Count 3 LargeStack
UserTime 00:00:00.000
KernelTime 00:00:00.000
Start Address 0x75b67cdf
Stack Init f80bd000 Current f80bc9c8 Base f80bd000 Limit f80ba000 Call 0
Priority 14 BasePriority 13 PriorityDecrement 0 DecrementCount 0
Kernel stack not resident.
ChildEBP RetAddr Args to Child
f80bc9e0 80500ce6 00000000 8246d998 804f9af2 nt!KiSwapContext+0x2e (FPO: [Uses EBP] [0,0,4])
f80bc9ec 804f9af2 804f986e e1627008 00000000 nt!KiSwapThread+0x46 (FPO: [0,0,0])
f80bca24 bf80a4a3 00000002 82475218 00000001 nt!KeWaitForMultipleObjects+0x284 (FPO: [Non-Fpo])
f80bca5c bf88c0a6 00000001 82475218 00000000 win32k!xxxMsgWaitForMultipleObjects+0xb0 (FPO: [Non-Fpo])
f80bcd30 bf87507d bf9ac0a0 00000001 f80bcd54 win32k!xxxDesktopThread+0x339 (FPO: [Non-Fpo])
f80bcd40 bf8010fd bf9ac0a0 f80bcd64 00bcfff4 win32k!xxxCreateSystemThreads+0x6a (FPO: [Non-Fpo])
f80bcd54 8053d648 00000000 00000022 00000000 win32k!NtUserCallOneParam+0x23 (FPO: [Non-Fpo])
f80bcd54 7c90e514 00000000 00000022 00000000 nt!KiFastCallEntry+0xf8 (FPO: [0,0] TrapFrame # f80bcd64)
This waitForMultipleObjects looks interesting because I'm wondering if csrss.exe is waiting on some event which isn't arriving to allow the logoff. Can anyone tell me how I might find out what event it's waiting for anything else I might do to further investigate the problem?
The objects being waited on are right there in the output:
THREAD 8246d998 Cid 0248.02a0 Teb: 7ffd7000 Win32Thread: e1627008 WAIT: (WrUserRequest) UserMode Non-Alertable
8243d9f0 SynchronizationEvent
81fe0390 SynchronizationEvent
I'll note though that the thread you're looking at is a common thread, just about every system that you look at will have it (not sure what that thread is for exactly, but I recognize the stack...Sometimes I feel like I've been doing this too long!).
I'll also note that you can't trust the parameters on the stack all of the time. See some details here: http://analyze-v.com/?p=7
-scott
To start off, try !object 82475218 to see if that tells you what the object is.
If that fails to help, try this:
http://blogs.msdn.com/search/SearchResults.aspx?q=KeWaitForMultipleObjects
It's a search for KeWaitForMultipleObjects on the NT Debugging Blog, which is a great blog in for learning about Windows internals.
EDIT:
Here's the documentation for KeWaitForMultipleObjects:
http://msdn.microsoft.com/en-us/library/ff553324.aspx
Cheers.
Jas.