What is the "Break instruction exception" in WinDbg? - windows

I'm debugging some random crash bugs, but actually very difficult to go deep into. Because when i open crash dump, only find one error:
0:000> .exr -1
ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
Actually i haven't set any hard-code breakpoint in code, so i search about this exception in google, some people said this exception may be caused by heap corruption.
So my question is,
Is there any other reason why cause this exception, except hard-code breakpoint, manual breakpoint while debugging, heap corruption?
Another question is, i try to use Application Verifier to check heap corruption, i understand how does it work, app verifier will trigger break instruction exception while heap corruption. But currently, i run without app verifier, who will raise the break instruction exception?
Additional info: call stack for current thread.
*0:000> k
ChildEBP RetAddr
0012f96c 7c827d19 ntdll!KiFastSystemCallRet
0012f970 77e6202c ntdll!NtWaitForMultipleObjects+0xc
0012fa18 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
0012fa74 3b288523 user32!RealMsgWaitForMultipleObjectsEx+0x141
0012fab8 3b32b9bd msenv!EnvironmentMsgLoop+0x1ea
0012fae4 3b32b94d msenv!CMsoCMHandler::FPushMessageLoop+0x86
0012fb0c 3b32b8e9 msenv!SCM::FPushMessageLoop+0xb7
0012fb28 3b32b8b8 msenv!SCM_MsoCompMgr::FPushMessageLoop+0x28
0012fb48 3b32be4e msenv!CMsoComponent::PushMsgLoop+0x28
0012fbe0 3b327561 msenv!VStudioMainLogged+0x482
0012fc0c 3000a4a6 msenv!VStudioMain+0xc1
0012fc38 30007301 devenv!util_CallVsMain+0xff
0012ff14 3000760c devenv!CDevEnvAppId::Run+0x91f
0012ff30 30007680 devenv!WinMain+0x74
0012ffc0 77e6f23b devenv!License::GetPID+0x258
0012fff0 00000000 kernel32!BaseProcessStart+0x23*
Our application is a Visual Studio Package.
Below is the result from !analyze -v
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
*** WARNING: Unable to verify checksum for mscorlib.ni.dll
*************************************************************************
*** ***
*** ***
*** Your debugger is not using the correct symbols ***
*** ***
*** In order for this command to work properly, your symbol path ***
*** must point to .pdb files that have full type information. ***
*** ***
*** Certain .pdb files (such as the public OS symbols) do not ***
*** contain the required information. Contact the group that ***
*** provided you with these symbols if you need this command to ***
*** work. ***
*** ***
*** Type referenced: kernel32!pNlsUserInfo ***
*** ***
*************************************************************************
*************************************************************************
*** ***
*** ***
*** Your debugger is not using the correct symbols ***
*** ***
*** In order for this command to work properly, your symbol path ***
*** must point to .pdb files that have full type information. ***
*** ***
*** Certain .pdb files (such as the public OS symbols) do not ***
*** contain the required information. Contact the group that ***
*** provided you with these symbols if you need this command to ***
*** work. ***
*** ***
*** Type referenced: kernel32!pNlsUserInfo ***
*** ***
*************************************************************************
FAULTING_IP:
+0
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
FAULTING_THREAD: 00001f1c
DEFAULT_BUCKET_ID: STATUS_BREAKPOINT
PROCESS_NAME: devenv.exe
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
MANAGED_STACK:
SP IP Function
0012E584 09C8A903 Microsoft_VisualStudio_Design!Microsoft.VisualStudio.NativeMethods.ThrowOnFailure(Int32, Int32[])+0x3b
0012E590 09C8C604 Microsoft_VisualStudio_Design!Microsoft.VisualStudio.Design.VSDesignSurfaceManager.Microsoft.VisualStudio.Shell.Interop.IVsSelectionEvents.OnElementValueChanged(UInt32, System.Object, System.Object)+0x144
StackTraceString: <none>
HResult: 80004005
EXCEPTION_OBJECT: !pe 3115d464
Exception object: 3115d464
Exception type: System.Runtime.InteropServices.COMException
Message: Error HRESULT E_FAIL has been returned from a call to a COM component.
InnerException: <none>
StackTrace (generated):
SP IP Function
0012E584 09C8A903 Microsoft_VisualStudio_Design!Microsoft.VisualStudio.NativeMethods.ThrowOnFailure(Int32, Int32[])+0x3b
0012E590 09C8C604 Microsoft_VisualStudio_Design!Microsoft.VisualStudio.Design.VSDesignSurfaceManager.Microsoft.VisualStudio.Shell.Interop.IVsSelectionEvents.OnElementValueChanged(UInt32, System.Object, System.Object)+0x144
StackTraceString: <none>
HResult: 80004005
MANAGED_OBJECT: !dumpobj 3201988
Name: System.String
MethodTable: 79330a00
EEClass: 790ed64c
Size: 158(0x9e) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: Error HRESULT E_FAIL has been returned from a call to a COM component.
Fields:
MT Field Offset Type VT Attr Value Name
79332c4c 4000096 4 System.Int32 1 instance 71 m_arrayLength
79332c4c 4000097 8 System.Int32 1 instance 70 m_stringLength
793316e0 4000098 c System.Char 1 instance 45 m_firstChar
79330a00 4000099 10 System.String 0 shared static Empty
>> Domain:Value 00219c28:03031198 <<
79331630 400009a 14 System.Char[] 0 shared static WhitespaceChars
>> Domain:Value 00219c28:03031798 <<
EXCEPTION_MESSAGE: Error HRESULT E_FAIL has been returned from a call to a COM component.
MANAGED_OBJECT_NAME: System.Runtime.InteropServices.COMException
LAST_CONTROL_TRANSFER: from 7c827d19 to 7c82860c
PRIMARY_PROBLEM_CLASS: STATUS_BREAKPOINT
BUGCHECK_STR: APPLICATION_FAULT_STATUS_BREAKPOINT
STACK_TEXT:
09c8a903 Microsoft_VisualStudio_Design!Microsoft.VisualStudio.NativeMethods.ThrowOnFailure
09c8c604 Microsoft_VisualStudio_Design!Microsoft.VisualStudio.Design.VSDesignSurfaceManager.Microsoft.VisualStudio.Shell.Interop.IVsSelectionEvents.OnElementValueChanged
STACK_COMMAND: dds 12e584 ; kb
FOLLOWUP_IP:
+9c8a903
09c8a903 8bc6 mov eax,esi
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: Microsoft_VisualStudio_Design!Microsoft.VisualStudio.NativeMethods.ThrowOnFailure+9c8a903
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: Microsoft_VisualStudio_Design
IMAGE_NAME: Microsoft.VisualStudio.Design.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 0
FAILURE_BUCKET_ID: STATUS_BREAKPOINT_80000003_Microsoft.VisualStudio.Design.dll!Microsoft.VisualStudio.NativeMethods.ThrowOnFailure
BUCKET_ID: APPLICATION_FAULT_STATUS_BREAKPOINT_Microsoft_VisualStudio_Design!Microsoft.VisualStudio.NativeMethods.ThrowOnFailure+9c8a903
Followup: MachineOwner
...
In the managed stack, there is a explicit error,Microsoft.VisualStudio.NativeMethods.ThrowOnFailure..
But that means the com exception cause the break instruction exception?
!analyze seems just dump the managed level, the com exception maybe the last error in the managed level.
I also search something about interrupt and exception from google, Normally, the break instruction exception can be triggered in following conditions:
1. Hardcode interrupt request, like: __asm int 3 (ASM), System.Diagnostics.Debugger.Break (C#), DebugBreak() (WinAPI).
2. OS enable memory runtime check, like Application Verifier can trigger after heap corruption, memory overrun.
3. Compiler can have some configuration to enble what should be filled for the uninitialized memory block and end of function(blank area, after retun..). For example, Microsoft VC complier can fill 0xCC if enable /GZ. 0xCC is actually a opcode of __asm int 3. So if some error cause the application run into such block, will trigger a break point.
Correct?
If that, I think Application Verifier should be best choice to find the root cause.

For future reference, the Your debugger is not using the correct symbols warning is caused because you need to add Windows symbols to the Windbg symbols path. Here is how to do that:
Set Microsoft symbol server path automatically:
0:000> .symfix
Optionally you can specify an additional location where to download symbol from, e.g.:
0:000> .sympath+ c:\myproject
Check current symbol search path:
0:000> .sympath
You should see something like this:
SRV**http://msdl.microsoft.com/download/symbols
Reload symbols:
0:000> .reload
Then, you will be able to see information about the current exception using this command:
0:000> !analyze -v
You should see a line similar to the following:
ExceptionCode: c0000005 (Access violation)
Good luck fixing bugs!

The command to use to find the exception that caused the crash dump is .ecxr. The outpt you got from .exr -1 is incorrect as the ExceptionAddress is zero.

Related

Just built j902 from source on osx, seeking confirmation that broken tests are "ok"

I have been using J for a few weeks and absolutely loving the new way it makes me look at old problems. I installed the ODBC drivers for Osx (10.15.6) but I get pointer violations:
[sean]% jconsole
load 'dd'
ddsrc''
jconsole(77340,0x1130efdc0) malloc: *** error for object 0x7f817ddcd1c0: pointer being freed was not allocated
jconsole(77340,0x1130efdc0) malloc: *** set a breakpoint in malloc_error_break to debug
[1] 77340 abort jconsole
[Seans-iMac:~]
[sean]%
I checked out the source from Github with the serious intent of trying to debug the problem for myself, I followed the instructions in the make.txt file, terse but mostly good; I had issues with the "./cpbin.sh" showing errors messages, and the first time I ran the tests it failed to find the .dylibs so I had to copy those manually
cp ../../bin/darwin/j64/*.dylib .
I then re-ran the tests:
sean#Seans-iMac:~/gitjsource/jsource/jlibrary/bin|master⚡ ⇒ ./jconsole ../../test/tsu.ijs
and his time the only errors were mach_vm_map errors, so a little progress but running RUN ddal gave this output which is superficially troubling!
RUN ddall
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=2251799813685248, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=2251799813685248, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=281474976710656, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
I then ran RECHO ddall and got this output:
RECHO ddall
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g0.ijs
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g000.ijs
:
: elided; assume to have passed, then...
:
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g210.ijs
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g210a.ijs
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=2251799813685248, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=2251799813685248, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=281474976710656, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g211.ijs
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g212.ijs
:
: elided; assume to have passed
:
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/gxco2.ijs
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/gxinf.ijs
0
I just want to know if this is --serious-- or can be ignored, I am off to look in failed test script g210a.ijs to check it out...all I wanted to do was try to see why ODBC 'ddsrc' blows chunks!
The tsu reports of "malloc_error_break" are normal. They indicate a malloc request that was expected to fail so things are OK. We don't know how to turn off those warnings. They do not occur on other platforms.
The same error in ddsrc does indicate a problem. Again it indicates a malloc that does not succeed. Probably because of a bug that gives a bad arg to malloc. This is not a problem in the J engine, but is a problem in the odbc interface.
It certainly looks serious, but the problem's not likely in jsource, but in the FFI code that you can find in addons/data/odbc/odbc.ijs or ddmysql. An excerpt:
select. UNAME
case. 'Linux' do. libodbc=: (0-:PREFER_IODBC){::'libiodbc.so.2';'libodbc.so.2'
case. 'Darwin' do. libodbc=: 'libiodbc.dylib'
case. 'Win' do. libodbc=: 'odbc32.dll'
case. do. libodbc=: 'libodbc.so'
end.
i.0 0
)
sqlallochandle=: (libodbc, ' SQLAllocHandle s s x *x') &cd
sqlbindcol=: (libodbc, ' SQLBindCol s x s s * x *') &cd
sqlbindparameter=: (libodbc, ' SQLBindParameter s x s s s s x s * i *') &cd
The problem could be
that you have a library sufficiently different from the library this was coded against (note it doesn't seem to be doing any version checking) that just trying to call into it is destroying memory and breaking j internals
that somewhere J is receiving what is erroneously treated as an owned pointer, and it's A) not a freeable pointer, or B) not a pointer freeable by J because the library is using its own allocator
that somewhere J is erroneously passing a pointer to the library that the library regards as an owned pointer that it should free, with similar problems, and the error message just deceptively sources the fault to jsource
In any case I suggest reporting this to the general# forum
Of the g210a.ijs errors, the file itself warns about errors that seem to fit what you're seeing, and which look to be unrelated to the odbc issue.
NB. Some tests fail because memory is obtained in 65536 word increments,
NB. and malloc does not always fail gracefully.
NB. Other tests fail because they try to grab the entire result array
NB. at the outset, and fail with a limit error or an out of memory

Unable to build MIT Kerberos in Windows 2007 (Windows Server Enterprise)

I was trying to build MIT Kerberos In Windows 2007 (Windows Server Enterprise) Service Pack 2 32 bit system. After adding a few flags specific to posix errors I was able to build it in Windows 7 (along with working kinit and klist programs). However in win 2007 all exes generated crash whenever I attempt to execute them. I had used Microsoft visual studio 2008 with Microsoft SDK v6 for both builds.
Crash code in event viewer: Exception code: 0xc000041d and occasionally 0xc00008c
Fault offset: 0x76e011f1
After enabling all possible checks in gflags and running kinit, I noticed a message saying unable to start application due to incorrect security permissions. I changed compatibility mode to xp3 and ran as administrator but no luck.
I then used sxstrace to determine any link time inconsistencies. I didnt find even a single line in my parsed trace file. I then used dependency walker and it wasnt able to find any errors.
I then used procdump and windbg to get the dump of the problem. Unfortunately I havent been able to locate a suitable pdb for nt.dll. This is what i found after attempting to unwind the core dump stack (kp command):-
0018975c 64754d57 user32!GetProcessWindowStation+0x15
0018a8c0 64755d08 msvcr90d!CrtDbgReport+0x437
0018f954 64754992 msvcr90d!VCrtDbgReportA+0x7d8
0018f974 6475494b msvcr90d!CrtDbgReport+0x72
0018f99c 646bc34d msvcr90d!CrtDbgReport+0x2b
0018f9d0 646bc812 msvcr90d!get_pgmptr+0x1bd
0018fa08 646bc711 msvcr90d!_getmainargs+0x182
0018fa1c 76fc99a0 msvcr90d!_getmainargs+0x81
0018fa3c 76fcd939 ntdll!RtlQueryEnvironmentVariable+0x241
0018fb30 76fd686c ntdll!LdrResSearchResource+0xb4d
0018fcb0 76fd5326 ntdll!RtlGetNtVersionNumbers+0x9b
0018fd00 76fc9ef9 ntdll!RtlSetUnhandledExceptionFilter+0x50
0018fd10 00000000 ntdll!LdrInitializeThunk+0x10
I dont quite understand what this means and I have no idea what on earth is going on. I dont have too much proficiency in using windbg
Is there anything else that anyone can suggest me to narrow down the root cause of the issue? Even after I copy the 2k7 built binaries to my local win 7 machine and it still crashes with the same stack.
Edit: after running .symfix, .reload and then analyze -v I got the following output in windbg console:-
*** WARNING: Unable to verify checksum for klist.exe
*** ERROR: Module load completed but symbols could not be loaded for klist.exe
FAULTING_IP:
+0
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
FAULTING_THREAD: 000014bc
PROCESS_NAME: klist.exe
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
APP: klist.exe
BUGCHECK_STR: APPLICATION_FAULT_STATUS_BREAKPOINT_AFTER_CALL
PRIMARY_PROBLEM_CLASS: STATUS_BREAKPOINT_AFTER_CALL
DEFAULT_BUCKET_ID: STATUS_BREAKPOINT_AFTER_CALL
LAST_CONTROL_TRANSFER: from 6475450f to 74c49eff
STACK_TEXT:
00189718 6475450f 0018973c 0018a8c0 64754cc0 user32!NtUserGetProcessWindowStation+0x15
0018975c 64754d57 001898b0 64696070 00012012 msvcr90d!__crtMessageBoxA+0x14f
0018a8c0 64755d08 00000001 00000000 00000000 msvcr90d!__crtMessageWindowA+0x3b7
0018f954 64754992 00000001 00000000 00000000 msvcr90d!_VCrtDbgReportA+0x7d8
0018f974 6475494b 00000001 00000000 00000000 msvcr90d!_CrtDbgReportV+0x22
0018f99c 646bc34d 00000001 00000000 00000000 msvcr90d!_CrtDbgReport+0x2b
0018f9d0 646bc812 00000022 6e76fe50 0018faec msvcr90d!_NMSG_WRITE+0x6d
0018fa08 646bc711 64680000 00000001 0018fd24 msvcr90d!__CRTDLL_INIT+0xf2
0018fa1c 76fc99a0 64680000 00000001 0018fd24 msvcr90d!_CRTDLL_INIT+0x21
0018fa3c 76fcd939 646bc6f0 64680000 00000001 ntdll!LdrpCallInitRoutine+0x14
0018fb30 76fd686c 0018fd24 7efdd000 7efde000 ntdll!LdrpRunInitializeRoutines+0x26f
0018fcb0 76fd5326 0018fd24 76f90000 734dc02c ntdll!LdrpInitializeProcess+0x1400
0018fd00 76fc9ef9 0018fd24 76f90000 00000000 ntdll!_LdrpInitialize+0x78
0018fd10 00000000 0018fd24 76f90000 00000000 ntdll!LdrInitializeThunk+0x10
FOLLOWUP_IP:
msvcr90d!__crtMessageBoxA+14f [f:\dd\vctools\crt_bld\self_x86\crt\src\crtmbox.c # 121]
6475450f 8945ec mov dword ptr [ebp-14h],eax
FAULTING_SOURCE_LINE: f:\dd\vctools\crt_bld\self_x86\crt\src\crtmbox.c
FAULTING_SOURCE_FILE: f:\dd\vctools\crt_bld\self_x86\crt\src\crtmbox.c
FAULTING_SOURCE_LINE_NUMBER: 121
SYMBOL_STACK_INDEX: 1
SYMBOL_NAME: msvcr90d!__crtMessageBoxA+14f
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: msvcr90d
IMAGE_NAME: msvcr90d.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 488ef6c7
STACK_COMMAND: dt ntdll!LdrpLastDllInitializer BaseDllName ; dt ntdll!LdrpFailureData ; ~0s; .ecxr ; kb
FAILURE_BUCKET_ID: STATUS_BREAKPOINT_AFTER_CALL_80000003_msvcr90d.dll!__crtMessageBoxA
BUCKET_ID: APPLICATION_FAULT_STATUS_BREAKPOINT_AFTER_CALL_msvcr90d!__crtMessageBoxA+14f
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/klist_exe/4_0_0_0/533e75fb/unknown/0_0_0_0/bbbbbbb4/80000003/00000000.htm?Retriage=1
Followup: MachineOwner
Edit: After running in Visual Studio I got the following output:-
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\klist.exe', Symbols loaded.
'klist.exe': Loaded 'C:\Windows\SysWOW64\ntdll.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\kernel32.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\KernelBase.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\sysfer.dll'
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\k5sprt32.dll', Symbols loaded.
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\msvcr90d.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\ws2_32.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\msvcrt.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\rpcrt4.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\sspicli.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\cryptbase.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\sechost.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\nsi.dll'
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\krb5_32.dll', Symbols loaded.
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\comerr32.dll', Symbols loaded.
'klist.exe': Loaded 'C:\Windows\SysWOW64\user32.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\gdi32.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\lpk.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\usp10.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\advapi32.dll'
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\wshelp32.dll', Symbols loaded.
'klist.exe': Loaded 'C:\Windows\SysWOW64\dnsapi.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\shell32.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\shlwapi.dll'
First-chance exception at 0x74c49eff in klist.exe: 0xC0000005: Access violation reading location 0x00000250.
*** An Access Violation occurred in "C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\klist.exe" :
The instruction at 0000000076E011F1 tried to read from an invalid address, 0000000000000250
*** enter .exr 000000000008E970 for the exception record
*** enter .cxr 000000000008E480 for the context
*** then kb to get the faulting stack
Unhandled exception at 0x74c49eff in klist.exe: 0xC000041D: An unhandled exception was encountered during a user callback.
> kb
Index Function
--------------------------------------------------------------------------------
*1 user32.dll!74c49eff()
2 [Frames below may be incorrect and/or missing, no symbols loaded for user32.dll]
3 user32.dll!74c49eff()
4 msvcr90d.dll!58f8450f()
5 msvcr90d.dll!58f84d57()
I cant get klist or krb5 dlls in the stack at all. Since klist or any other mit kerb dll does not appear in this section, I am unable to load check their symbols. This is very frustrating, I will attempt to build my own sample program and check for issues. Btw did I miss any analysis steps?
Edit : After checking for first argument to crtmessagebox I got :-
001898b0 "Debug Error!..Program: C:\WS\TPL"
001898d0 "\src\MitKerberos\1.11.1\BUILDDEB"
001898f0 "UG\bin\klist.exe..R6034..An appl"
00189910 "ication has made an attempt to l"
00189930 "oad the C runtime library withou"
00189950 "t using a manifest..This is an u"
00189970 "nsupported way to load Visual C+"
00189990 "+ DLLs. You need to modify your "
001899b0 "application to build with a mani"
001899d0 "fest..For more information, see "
001899f0 "the "Visual C++ Libraries as Sha"
00189a10 "red Side-by-Side Assemblies" top"
As far as I understand the program responsible for this is mt.exe and I had run it.

VB6 Crash Dump Symbol is not being resolved

I am unable to figure this problem out. Symbol is not being resolved
Deployment
There are number of exes of my system deployed on a network path. All users run those exes from that shared network path. This was working fine two weeks ago but now some of those exes have started crashing. There is no fix pattern of being crashed, it happens to any user, anytime during any activity.
Troubleshooting
I have got the dump of one of them, i tried WinDbg and got following
Microsoft (R) Windows Debugger Version 6.2.9200.16384 X86
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\crash\RNS1000.exe.mdmp]
User Mini Dump File: Only registers, stack and portions of memory are available
Symbol search path is: SRV*c:\crash*http://msdl.microsoft.com/download/symbols;c:\crash
Executable search path is:
Windows XP Version 2600 (Service Pack 3) MP (2 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
Machine Name:
Debug session time: Wed Oct 10 15:36:36.000 2012 (UTC + 5:00)
System Uptime: not available
Process Uptime: 0 days 7:12:54.000
................................................................
.........................................................
Loading unloaded module list
.......
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(924.85c): In-page I/O error ffffffffc000020c - code c0000006 (first/second chance not available)
eax=02060000 ebx=7c90fe01 ecx=00001000 edx=7c90e4f4 esi=000003a0 edi=00000000
eip=7c90e4f4 esp=0013afdc ebp=0013b040 iopl=0 nv up ei ng nz ac pe cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00200297
ntdll!KiFastSystemCallRet:
7c90e4f4 c3 ret
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
GetUrlPageData2 (WinHttp) failed: 12007.
FAULTING_IP:
RNS1000+55f610
0095f610 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0095f610 (RNS1000+0x0055f610)
ExceptionCode: c0000006 (In-page I/O error)
ExceptionFlags: 00000000
NumberParameters: 3
Parameter[0]: 00000008
Parameter[1]: 0095f610
Parameter[2]: c000020c
Inpage operation failed at 0095f610, due to I/O error c000020c
DEFAULT_BUCKET_ID: SOFTWARE_NX_FAULT
PROCESS_NAME: RNS1000.exe
ERROR_CODE: (NTSTATUS) 0xc0000006 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The required data was not placed into memory because of an I/O error status of "0x%08lx".
EXCEPTION_CODE: (NTSTATUS) 0xc0000006 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The required data was not placed into memory because of an I/O error status of "0x%08lx".
EXCEPTION_PARAMETER1: 00000008
EXCEPTION_PARAMETER2: 0095f610
EXCEPTION_PARAMETER3: c000020c
IO_ERROR: (NTSTATUS) 0xc000020c - The transport connection is now disconnected.
ADDITIONAL_DEBUG_TEXT: Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]
LAST_CONTROL_TRANSFER: from 00000000 to 00000000
APP: rns1000.exe
FAULTING_THREAD: ffffffff
PRIMARY_PROBLEM_CLASS: SOFTWARE_NX_FAULT
BUGCHECK_STR: APPLICATION_FAULT_SOFTWARE_NX_FAULT
STACK_TEXT:
00000000 00000000 hardware_disk!Unknown+0x0
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: hardware_disk!Unknown
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: hardware_disk
DEBUG_FLR_IMAGE_TIMESTAMP: 0
STACK_COMMAND: ** Pseudo Context ** ; kb
FAILURE_BUCKET_ID: SOFTWARE_NX_FAULT_c0000006_hardware_disk!Unknown
BUCKET_ID: APPLICATION_FAULT_SOFTWARE_NX_FAULT_hardware_disk!Unknown
IMAGE_NAME: hardware_disk
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/RNS1000_exe/2_0_0_5/4f17b9d2/RNS1000_exe/2_0_0_5/4f17b9d2/c0000006/0055f610.htm?Retriage=1
Followup: MachineOwner
---------
I am expecting RNS1000+55f610 to be resolved to one of my programs function but it has not been resolved. The sysmbol path contains exe, pdb and mdmp.
Please tell me why has it not been resolved? what wrong am i doing?
The key part here is the In-page I/O error. The underlying disk/network drive disappeared.
The crash occurs some time later when it tries to page back in part of the executable, but it no longer has a valid file handle/connection.
The only fix is to run it locally or make sure the disk doesn't disappear while they're running.
More generally, you can get VB to create the info files for native debugging using the "Create symbolic debug info" option in the project's Compile settings. This can only be done before the fact though and won't help with debugging an existing build.

Windows application debugging

My operations group, and the devs are clueless to why this happens. Basically the server-side service of a client/server application hangs. We have been pushing new bugfixed versions and providing all kinds of logs to the devs, but they can't figure it out. To make it even harder to figure out, this is an application that works very closely with another application on the client side.
I have no idea if I'm onto something here, as I have limited windbg experience, but this seems worth checking out. Google comes up pretty promising, but with mostly gamers having BSODs not providing much more info...
I found this bit at the start of the dump:
WARNING: odbccp32 overlaps comctl32 .
WARNING: odbc32 overlaps odbccp32
WARNING: odbc32 overlaps comctl32 .............
WARNING: mswsock overlaps FWPUCLNT .......
WARNING: winsta overlaps winnsi .
WARNING: ntlanman overlaps drprov .... ...
WARNING: srvcli overlaps netapi32
WARNING: wkscli overlaps srvcli ..........
WARNING: ncrypt overlaps schannel .
WARNING: nlaapi overlaps ncrypt .
WARNING: NapiNSP overlaps nlaapi ....
WARNING: rsaenh overlaps cryptsp
and a bit lower:
OVERLAPPED_MODULE: Address regions for 'odbc32' and 'odbccp32' overlap
Here is the complete !analyze -v dump:
Microsoft (R) Windows Debugger Version 6.2.8400.0 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\debug\MES\PLSMES.dmp]
User Mini Dump File: Only registers, stack and portions of memory are available
Symbol search path is: srv*c:\symbols*c:\windows\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (16 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Machine Name:
Debug session time: Thu Jun 14 10:37:01.000 2012 (UTC + 2:00)
System Uptime: not available
Process Uptime: 0 days 6:36:13.000
......................................WARNING: odbccp32 overlaps comctl32
.WARNING: odbc32 overlaps odbccp32
WARNING: odbc32 overlaps comctl32
.............WARNING: mswsock overlaps FWPUCLNT
.......WARNING: winsta overlaps winnsi
.WARNING: ntlanman overlaps drprov
....
...WARNING: srvcli overlaps netapi32
.WARNING: wkscli overlaps srvcli
..........WARNING: ncrypt overlaps schannel
.WARNING: nlaapi overlaps ncrypt
.WARNING: NapiNSP overlaps nlaapi
....WARNING: rsaenh overlaps cryptsp
Cannot read PEB32 from WOW64 TEB32 7efdd000 - Win32 error 0n30
wow64cpu!CpupSyscallStub+0x9:
00000000`741f2e09 c3 ret
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
*** WARNING: Unable to verify timestamp for PLSMES.exe
*** ERROR: Module load completed but symbols could not be loaded for PLSMES.exe
FAULTING_IP:
+0
00000000`00000000 ?? ???
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000000000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
FAULTING_THREAD: 0000000000001364
DEFAULT_BUCKET_ID: BAD_DUMP_MISSING_MEMORY
PROCESS_NAME: PLSMES.exe
OVERLAPPED_MODULE: Address regions for 'odbc32' and 'odbccp32' overlap
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
APP: plsmes.exe
ADDITIONAL_DEBUG_TEXT: Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]
LAST_CONTROL_TRANSFER: from 0000000000000000 to 00000000741f2e09
PRIMARY_PROBLEM_CLASS: BAD_DUMP_MISSING_MEMORY
BUGCHECK_STR: APPLICATION_FAULT_BAD_DUMP_MISSING_MEMORY
STACK_TEXT:
00000000`00000000 00000000`00000000 bad_dump!missing_stack+0x0
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: bad_dump!missing_stack
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: bad_dump
IMAGE_NAME: bad_dump
DEBUG_FLR_IMAGE_TIMESTAMP: 0
STACK_COMMAND: ** Pseudo Context ** ; kb
FAILURE_BUCKET_ID: BAD_DUMP_MISSING_MEMORY_80000003_bad_dump!missing_stack
BUCKET_ID: X64_APPLICATION_FAULT_BAD_DUMP_MISSING_MEMORY_bad_dump!missing_stack
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/PLSMES_exe/4_4_3_2582/4f8ac8f6/unknown/0_0_0_0/bbbbbbb4/80000003/00000000.htm?Retriage=1
Followup: MachineOwner
---------
Could this be related to the application hangs? Overlapping memory doesn't seem good.
Note: The same server runs other instances of the same application without error.
Any pointers to further debugging would also be nice.
(Moved from ServerFault, I guess this is better asked here.)
I've seen this happen when a 64 bit version of task manager is used to create a dump of a 32 bit process. If this is your case, then use the 32 bit version of task manager which can be found in the SysWOW64 folder. This link describes the problem:
http://blogs.msdn.com/b/tess/archive/2010/09/29/capturing-memory-dumps-for-32-bit-processes-on-an-x64-machine.aspx

Windbg work so slowly?

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(10e0.cd8): Wake debugger - code 80000007 (first/second chance not available)
eax=7c9bc668 ebx=00000000 ecx=00000000 edx=7c9bc080 esi=00000198 edi=00000000
eip=7c95847c esp=0006fc0c ebp=0006fc7c iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
ntdll!KiFastSystemCallRet:
7c95847c c3 ret
0:000> .ecxr
eax=7c9bc668 ebx=00000000 ecx=00000000 edx=7c9bc080 esi=00000198 edi=00000000
eip=7c95847c esp=0006fc0c ebp=0006fc7c iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
ntdll!KiFastSystemCallRet:
7c95847c c3 ret
0:000> kb
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr Args to Child
0006fc08 7c957b99 7c821d1e 00000198 00000000 ntdll!KiFastSystemCallRet
*** ERROR: Symbol file could not be found. Defaulted to export symbols for kernel32.dll -
0006fc0c 7c821d1e 00000198 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
0006fc7c 7c821c8d 00000198 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0x88
*** ERROR: Symbol file could not be found. Defaulted to export symbols for w3dt.dll -
0006fc90 5a604692 00000198 ffffffff 00000000 kernel32!WaitForSingleObject+0x12
*** ERROR: Module load completed but symbols could not be loaded for w3wp.exe
0006ff0c 0100187c 00000007 002e3bd8 00000000 w3dt!IPM_MESSAGE_PIPE::operator=+0x1b31
0006ff44 01001a27 00000007 002e3bd8 002e46d8 w3wp+0x187c
0006ffc0 7c82f23b 00000000 00000000 7ffdf000 w3wp+0x1a27
0006fff0 00000000 010018f8 00000000 78746341 kernel32!ProcessIdToSessionId+0x209
load the symbols...and just wait....use the common command,like the '!threads',
and it works so slowly,here is the kb result,how to solve this problem?
waitting for a long time,and it shows:
0:000> .load sos
0:000> !threadpool
*** ERROR: Symbol file could not be found. Defaulted to export symbols for mscorwks.dll -
PDB symbol for mscorwks.dll not loaded
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of mscorwks.dll is
in the version directory
3) or, if you are debugging a dump file, verify that the file
mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file.
For example, an IA64 dump file must be debugged on an IA64
machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to mscorwks.dll as well.
so what's the problem?
Is your Internet connection slow?
WinDBG would fetch some symbol images from Mircosoft's site if they are not locally available. It occurs to me once that windbg almost freezes when loading some symbols because of my slow Internet connection.
Well, windbg is slow. period :)
You can try and use local symbols , but if you are using the MS symbol server be ready for some slow debugging.

Resources