I have been using J for a few weeks and absolutely loving the new way it makes me look at old problems. I installed the ODBC drivers for Osx (10.15.6) but I get pointer violations:
[sean]% jconsole
load 'dd'
ddsrc''
jconsole(77340,0x1130efdc0) malloc: *** error for object 0x7f817ddcd1c0: pointer being freed was not allocated
jconsole(77340,0x1130efdc0) malloc: *** set a breakpoint in malloc_error_break to debug
[1] 77340 abort jconsole
[Seans-iMac:~]
[sean]%
I checked out the source from Github with the serious intent of trying to debug the problem for myself, I followed the instructions in the make.txt file, terse but mostly good; I had issues with the "./cpbin.sh" showing errors messages, and the first time I ran the tests it failed to find the .dylibs so I had to copy those manually
cp ../../bin/darwin/j64/*.dylib .
I then re-ran the tests:
sean#Seans-iMac:~/gitjsource/jsource/jlibrary/bin|master⚡ ⇒ ./jconsole ../../test/tsu.ijs
and his time the only errors were mach_vm_map errors, so a little progress but running RUN ddal gave this output which is superficially troubling!
RUN ddall
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=2251799813685248, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=2251799813685248, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=281474976710656, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
I then ran RECHO ddall and got this output:
RECHO ddall
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g0.ijs
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g000.ijs
:
: elided; assume to have passed, then...
:
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g210.ijs
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g210a.ijs
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=2251799813685248, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=2251799813685248, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=281474976710656, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=562949953421312, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
jconsole(76285,0x1152c2dc0) malloc: can't allocate region
:*** mach_vm_map(size=1125899906842624, flags: 100) failed (error code=3)
jconsole(76285,0x1152c2dc0) malloc: *** set a breakpoint in malloc_error_break to debug
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g211.ijs
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/g212.ijs
:
: elided; assume to have passed
:
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/gxco2.ijs
/Users/sean/gitjsource/jsource/jlibrary/bin/../../test/gxinf.ijs
0
I just want to know if this is --serious-- or can be ignored, I am off to look in failed test script g210a.ijs to check it out...all I wanted to do was try to see why ODBC 'ddsrc' blows chunks!
The tsu reports of "malloc_error_break" are normal. They indicate a malloc request that was expected to fail so things are OK. We don't know how to turn off those warnings. They do not occur on other platforms.
The same error in ddsrc does indicate a problem. Again it indicates a malloc that does not succeed. Probably because of a bug that gives a bad arg to malloc. This is not a problem in the J engine, but is a problem in the odbc interface.
It certainly looks serious, but the problem's not likely in jsource, but in the FFI code that you can find in addons/data/odbc/odbc.ijs or ddmysql. An excerpt:
select. UNAME
case. 'Linux' do. libodbc=: (0-:PREFER_IODBC){::'libiodbc.so.2';'libodbc.so.2'
case. 'Darwin' do. libodbc=: 'libiodbc.dylib'
case. 'Win' do. libodbc=: 'odbc32.dll'
case. do. libodbc=: 'libodbc.so'
end.
i.0 0
)
sqlallochandle=: (libodbc, ' SQLAllocHandle s s x *x') &cd
sqlbindcol=: (libodbc, ' SQLBindCol s x s s * x *') &cd
sqlbindparameter=: (libodbc, ' SQLBindParameter s x s s s s x s * i *') &cd
The problem could be
that you have a library sufficiently different from the library this was coded against (note it doesn't seem to be doing any version checking) that just trying to call into it is destroying memory and breaking j internals
that somewhere J is receiving what is erroneously treated as an owned pointer, and it's A) not a freeable pointer, or B) not a pointer freeable by J because the library is using its own allocator
that somewhere J is erroneously passing a pointer to the library that the library regards as an owned pointer that it should free, with similar problems, and the error message just deceptively sources the fault to jsource
In any case I suggest reporting this to the general# forum
Of the g210a.ijs errors, the file itself warns about errors that seem to fit what you're seeing, and which look to be unrelated to the odbc issue.
NB. Some tests fail because memory is obtained in 65536 word increments,
NB. and malloc does not always fail gracefully.
NB. Other tests fail because they try to grab the entire result array
NB. at the outset, and fail with a limit error or an out of memory
I was trying to build MIT Kerberos In Windows 2007 (Windows Server Enterprise) Service Pack 2 32 bit system. After adding a few flags specific to posix errors I was able to build it in Windows 7 (along with working kinit and klist programs). However in win 2007 all exes generated crash whenever I attempt to execute them. I had used Microsoft visual studio 2008 with Microsoft SDK v6 for both builds.
Crash code in event viewer: Exception code: 0xc000041d and occasionally 0xc00008c
Fault offset: 0x76e011f1
After enabling all possible checks in gflags and running kinit, I noticed a message saying unable to start application due to incorrect security permissions. I changed compatibility mode to xp3 and ran as administrator but no luck.
I then used sxstrace to determine any link time inconsistencies. I didnt find even a single line in my parsed trace file. I then used dependency walker and it wasnt able to find any errors.
I then used procdump and windbg to get the dump of the problem. Unfortunately I havent been able to locate a suitable pdb for nt.dll. This is what i found after attempting to unwind the core dump stack (kp command):-
0018975c 64754d57 user32!GetProcessWindowStation+0x15
0018a8c0 64755d08 msvcr90d!CrtDbgReport+0x437
0018f954 64754992 msvcr90d!VCrtDbgReportA+0x7d8
0018f974 6475494b msvcr90d!CrtDbgReport+0x72
0018f99c 646bc34d msvcr90d!CrtDbgReport+0x2b
0018f9d0 646bc812 msvcr90d!get_pgmptr+0x1bd
0018fa08 646bc711 msvcr90d!_getmainargs+0x182
0018fa1c 76fc99a0 msvcr90d!_getmainargs+0x81
0018fa3c 76fcd939 ntdll!RtlQueryEnvironmentVariable+0x241
0018fb30 76fd686c ntdll!LdrResSearchResource+0xb4d
0018fcb0 76fd5326 ntdll!RtlGetNtVersionNumbers+0x9b
0018fd00 76fc9ef9 ntdll!RtlSetUnhandledExceptionFilter+0x50
0018fd10 00000000 ntdll!LdrInitializeThunk+0x10
I dont quite understand what this means and I have no idea what on earth is going on. I dont have too much proficiency in using windbg
Is there anything else that anyone can suggest me to narrow down the root cause of the issue? Even after I copy the 2k7 built binaries to my local win 7 machine and it still crashes with the same stack.
Edit: after running .symfix, .reload and then analyze -v I got the following output in windbg console:-
*** WARNING: Unable to verify checksum for klist.exe
*** ERROR: Module load completed but symbols could not be loaded for klist.exe
FAULTING_IP:
+0
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
FAULTING_THREAD: 000014bc
PROCESS_NAME: klist.exe
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
APP: klist.exe
BUGCHECK_STR: APPLICATION_FAULT_STATUS_BREAKPOINT_AFTER_CALL
PRIMARY_PROBLEM_CLASS: STATUS_BREAKPOINT_AFTER_CALL
DEFAULT_BUCKET_ID: STATUS_BREAKPOINT_AFTER_CALL
LAST_CONTROL_TRANSFER: from 6475450f to 74c49eff
STACK_TEXT:
00189718 6475450f 0018973c 0018a8c0 64754cc0 user32!NtUserGetProcessWindowStation+0x15
0018975c 64754d57 001898b0 64696070 00012012 msvcr90d!__crtMessageBoxA+0x14f
0018a8c0 64755d08 00000001 00000000 00000000 msvcr90d!__crtMessageWindowA+0x3b7
0018f954 64754992 00000001 00000000 00000000 msvcr90d!_VCrtDbgReportA+0x7d8
0018f974 6475494b 00000001 00000000 00000000 msvcr90d!_CrtDbgReportV+0x22
0018f99c 646bc34d 00000001 00000000 00000000 msvcr90d!_CrtDbgReport+0x2b
0018f9d0 646bc812 00000022 6e76fe50 0018faec msvcr90d!_NMSG_WRITE+0x6d
0018fa08 646bc711 64680000 00000001 0018fd24 msvcr90d!__CRTDLL_INIT+0xf2
0018fa1c 76fc99a0 64680000 00000001 0018fd24 msvcr90d!_CRTDLL_INIT+0x21
0018fa3c 76fcd939 646bc6f0 64680000 00000001 ntdll!LdrpCallInitRoutine+0x14
0018fb30 76fd686c 0018fd24 7efdd000 7efde000 ntdll!LdrpRunInitializeRoutines+0x26f
0018fcb0 76fd5326 0018fd24 76f90000 734dc02c ntdll!LdrpInitializeProcess+0x1400
0018fd00 76fc9ef9 0018fd24 76f90000 00000000 ntdll!_LdrpInitialize+0x78
0018fd10 00000000 0018fd24 76f90000 00000000 ntdll!LdrInitializeThunk+0x10
FOLLOWUP_IP:
msvcr90d!__crtMessageBoxA+14f [f:\dd\vctools\crt_bld\self_x86\crt\src\crtmbox.c # 121]
6475450f 8945ec mov dword ptr [ebp-14h],eax
FAULTING_SOURCE_LINE: f:\dd\vctools\crt_bld\self_x86\crt\src\crtmbox.c
FAULTING_SOURCE_FILE: f:\dd\vctools\crt_bld\self_x86\crt\src\crtmbox.c
FAULTING_SOURCE_LINE_NUMBER: 121
SYMBOL_STACK_INDEX: 1
SYMBOL_NAME: msvcr90d!__crtMessageBoxA+14f
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: msvcr90d
IMAGE_NAME: msvcr90d.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 488ef6c7
STACK_COMMAND: dt ntdll!LdrpLastDllInitializer BaseDllName ; dt ntdll!LdrpFailureData ; ~0s; .ecxr ; kb
FAILURE_BUCKET_ID: STATUS_BREAKPOINT_AFTER_CALL_80000003_msvcr90d.dll!__crtMessageBoxA
BUCKET_ID: APPLICATION_FAULT_STATUS_BREAKPOINT_AFTER_CALL_msvcr90d!__crtMessageBoxA+14f
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/klist_exe/4_0_0_0/533e75fb/unknown/0_0_0_0/bbbbbbb4/80000003/00000000.htm?Retriage=1
Followup: MachineOwner
Edit: After running in Visual Studio I got the following output:-
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\klist.exe', Symbols loaded.
'klist.exe': Loaded 'C:\Windows\SysWOW64\ntdll.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\kernel32.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\KernelBase.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\sysfer.dll'
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\k5sprt32.dll', Symbols loaded.
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\msvcr90d.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\ws2_32.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\msvcrt.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\rpcrt4.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\sspicli.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\cryptbase.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\sechost.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\nsi.dll'
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\krb5_32.dll', Symbols loaded.
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\comerr32.dll', Symbols loaded.
'klist.exe': Loaded 'C:\Windows\SysWOW64\user32.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\gdi32.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\lpk.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\usp10.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\advapi32.dll'
'klist.exe': Loaded 'C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\wshelp32.dll', Symbols loaded.
'klist.exe': Loaded 'C:\Windows\SysWOW64\dnsapi.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\shell32.dll'
'klist.exe': Loaded 'C:\Windows\SysWOW64\shlwapi.dll'
First-chance exception at 0x74c49eff in klist.exe: 0xC0000005: Access violation reading location 0x00000250.
*** An Access Violation occurred in "C:\WS\TPL\src\MitKerberos\1.11.1\BUILDDEBUG\bin\klist.exe" :
The instruction at 0000000076E011F1 tried to read from an invalid address, 0000000000000250
*** enter .exr 000000000008E970 for the exception record
*** enter .cxr 000000000008E480 for the context
*** then kb to get the faulting stack
Unhandled exception at 0x74c49eff in klist.exe: 0xC000041D: An unhandled exception was encountered during a user callback.
> kb
Index Function
--------------------------------------------------------------------------------
*1 user32.dll!74c49eff()
2 [Frames below may be incorrect and/or missing, no symbols loaded for user32.dll]
3 user32.dll!74c49eff()
4 msvcr90d.dll!58f8450f()
5 msvcr90d.dll!58f84d57()
I cant get klist or krb5 dlls in the stack at all. Since klist or any other mit kerb dll does not appear in this section, I am unable to load check their symbols. This is very frustrating, I will attempt to build my own sample program and check for issues. Btw did I miss any analysis steps?
Edit : After checking for first argument to crtmessagebox I got :-
001898b0 "Debug Error!..Program: C:\WS\TPL"
001898d0 "\src\MitKerberos\1.11.1\BUILDDEB"
001898f0 "UG\bin\klist.exe..R6034..An appl"
00189910 "ication has made an attempt to l"
00189930 "oad the C runtime library withou"
00189950 "t using a manifest..This is an u"
00189970 "nsupported way to load Visual C+"
00189990 "+ DLLs. You need to modify your "
001899b0 "application to build with a mani"
001899d0 "fest..For more information, see "
001899f0 "the "Visual C++ Libraries as Sha"
00189a10 "red Side-by-Side Assemblies" top"
As far as I understand the program responsible for this is mt.exe and I had run it.
I am unable to figure this problem out. Symbol is not being resolved
Deployment
There are number of exes of my system deployed on a network path. All users run those exes from that shared network path. This was working fine two weeks ago but now some of those exes have started crashing. There is no fix pattern of being crashed, it happens to any user, anytime during any activity.
Troubleshooting
I have got the dump of one of them, i tried WinDbg and got following
Microsoft (R) Windows Debugger Version 6.2.9200.16384 X86
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\crash\RNS1000.exe.mdmp]
User Mini Dump File: Only registers, stack and portions of memory are available
Symbol search path is: SRV*c:\crash*http://msdl.microsoft.com/download/symbols;c:\crash
Executable search path is:
Windows XP Version 2600 (Service Pack 3) MP (2 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
Machine Name:
Debug session time: Wed Oct 10 15:36:36.000 2012 (UTC + 5:00)
System Uptime: not available
Process Uptime: 0 days 7:12:54.000
................................................................
.........................................................
Loading unloaded module list
.......
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(924.85c): In-page I/O error ffffffffc000020c - code c0000006 (first/second chance not available)
eax=02060000 ebx=7c90fe01 ecx=00001000 edx=7c90e4f4 esi=000003a0 edi=00000000
eip=7c90e4f4 esp=0013afdc ebp=0013b040 iopl=0 nv up ei ng nz ac pe cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00200297
ntdll!KiFastSystemCallRet:
7c90e4f4 c3 ret
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
GetUrlPageData2 (WinHttp) failed: 12007.
FAULTING_IP:
RNS1000+55f610
0095f610 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0095f610 (RNS1000+0x0055f610)
ExceptionCode: c0000006 (In-page I/O error)
ExceptionFlags: 00000000
NumberParameters: 3
Parameter[0]: 00000008
Parameter[1]: 0095f610
Parameter[2]: c000020c
Inpage operation failed at 0095f610, due to I/O error c000020c
DEFAULT_BUCKET_ID: SOFTWARE_NX_FAULT
PROCESS_NAME: RNS1000.exe
ERROR_CODE: (NTSTATUS) 0xc0000006 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The required data was not placed into memory because of an I/O error status of "0x%08lx".
EXCEPTION_CODE: (NTSTATUS) 0xc0000006 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The required data was not placed into memory because of an I/O error status of "0x%08lx".
EXCEPTION_PARAMETER1: 00000008
EXCEPTION_PARAMETER2: 0095f610
EXCEPTION_PARAMETER3: c000020c
IO_ERROR: (NTSTATUS) 0xc000020c - The transport connection is now disconnected.
ADDITIONAL_DEBUG_TEXT: Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]
LAST_CONTROL_TRANSFER: from 00000000 to 00000000
APP: rns1000.exe
FAULTING_THREAD: ffffffff
PRIMARY_PROBLEM_CLASS: SOFTWARE_NX_FAULT
BUGCHECK_STR: APPLICATION_FAULT_SOFTWARE_NX_FAULT
STACK_TEXT:
00000000 00000000 hardware_disk!Unknown+0x0
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: hardware_disk!Unknown
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: hardware_disk
DEBUG_FLR_IMAGE_TIMESTAMP: 0
STACK_COMMAND: ** Pseudo Context ** ; kb
FAILURE_BUCKET_ID: SOFTWARE_NX_FAULT_c0000006_hardware_disk!Unknown
BUCKET_ID: APPLICATION_FAULT_SOFTWARE_NX_FAULT_hardware_disk!Unknown
IMAGE_NAME: hardware_disk
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/RNS1000_exe/2_0_0_5/4f17b9d2/RNS1000_exe/2_0_0_5/4f17b9d2/c0000006/0055f610.htm?Retriage=1
Followup: MachineOwner
---------
I am expecting RNS1000+55f610 to be resolved to one of my programs function but it has not been resolved. The sysmbol path contains exe, pdb and mdmp.
Please tell me why has it not been resolved? what wrong am i doing?
The key part here is the In-page I/O error. The underlying disk/network drive disappeared.
The crash occurs some time later when it tries to page back in part of the executable, but it no longer has a valid file handle/connection.
The only fix is to run it locally or make sure the disk doesn't disappear while they're running.
More generally, you can get VB to create the info files for native debugging using the "Create symbolic debug info" option in the project's Compile settings. This can only be done before the fact though and won't help with debugging an existing build.
My operations group, and the devs are clueless to why this happens. Basically the server-side service of a client/server application hangs. We have been pushing new bugfixed versions and providing all kinds of logs to the devs, but they can't figure it out. To make it even harder to figure out, this is an application that works very closely with another application on the client side.
I have no idea if I'm onto something here, as I have limited windbg experience, but this seems worth checking out. Google comes up pretty promising, but with mostly gamers having BSODs not providing much more info...
I found this bit at the start of the dump:
WARNING: odbccp32 overlaps comctl32 .
WARNING: odbc32 overlaps odbccp32
WARNING: odbc32 overlaps comctl32 .............
WARNING: mswsock overlaps FWPUCLNT .......
WARNING: winsta overlaps winnsi .
WARNING: ntlanman overlaps drprov .... ...
WARNING: srvcli overlaps netapi32
WARNING: wkscli overlaps srvcli ..........
WARNING: ncrypt overlaps schannel .
WARNING: nlaapi overlaps ncrypt .
WARNING: NapiNSP overlaps nlaapi ....
WARNING: rsaenh overlaps cryptsp
and a bit lower:
OVERLAPPED_MODULE: Address regions for 'odbc32' and 'odbccp32' overlap
Here is the complete !analyze -v dump:
Microsoft (R) Windows Debugger Version 6.2.8400.0 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\debug\MES\PLSMES.dmp]
User Mini Dump File: Only registers, stack and portions of memory are available
Symbol search path is: srv*c:\symbols*c:\windows\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (16 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Machine Name:
Debug session time: Thu Jun 14 10:37:01.000 2012 (UTC + 2:00)
System Uptime: not available
Process Uptime: 0 days 6:36:13.000
......................................WARNING: odbccp32 overlaps comctl32
.WARNING: odbc32 overlaps odbccp32
WARNING: odbc32 overlaps comctl32
.............WARNING: mswsock overlaps FWPUCLNT
.......WARNING: winsta overlaps winnsi
.WARNING: ntlanman overlaps drprov
....
...WARNING: srvcli overlaps netapi32
.WARNING: wkscli overlaps srvcli
..........WARNING: ncrypt overlaps schannel
.WARNING: nlaapi overlaps ncrypt
.WARNING: NapiNSP overlaps nlaapi
....WARNING: rsaenh overlaps cryptsp
Cannot read PEB32 from WOW64 TEB32 7efdd000 - Win32 error 0n30
wow64cpu!CpupSyscallStub+0x9:
00000000`741f2e09 c3 ret
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
*** WARNING: Unable to verify timestamp for PLSMES.exe
*** ERROR: Module load completed but symbols could not be loaded for PLSMES.exe
FAULTING_IP:
+0
00000000`00000000 ?? ???
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000000000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
FAULTING_THREAD: 0000000000001364
DEFAULT_BUCKET_ID: BAD_DUMP_MISSING_MEMORY
PROCESS_NAME: PLSMES.exe
OVERLAPPED_MODULE: Address regions for 'odbc32' and 'odbccp32' overlap
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
APP: plsmes.exe
ADDITIONAL_DEBUG_TEXT: Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]
LAST_CONTROL_TRANSFER: from 0000000000000000 to 00000000741f2e09
PRIMARY_PROBLEM_CLASS: BAD_DUMP_MISSING_MEMORY
BUGCHECK_STR: APPLICATION_FAULT_BAD_DUMP_MISSING_MEMORY
STACK_TEXT:
00000000`00000000 00000000`00000000 bad_dump!missing_stack+0x0
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: bad_dump!missing_stack
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: bad_dump
IMAGE_NAME: bad_dump
DEBUG_FLR_IMAGE_TIMESTAMP: 0
STACK_COMMAND: ** Pseudo Context ** ; kb
FAILURE_BUCKET_ID: BAD_DUMP_MISSING_MEMORY_80000003_bad_dump!missing_stack
BUCKET_ID: X64_APPLICATION_FAULT_BAD_DUMP_MISSING_MEMORY_bad_dump!missing_stack
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/PLSMES_exe/4_4_3_2582/4f8ac8f6/unknown/0_0_0_0/bbbbbbb4/80000003/00000000.htm?Retriage=1
Followup: MachineOwner
---------
Could this be related to the application hangs? Overlapping memory doesn't seem good.
Note: The same server runs other instances of the same application without error.
Any pointers to further debugging would also be nice.
(Moved from ServerFault, I guess this is better asked here.)
I've seen this happen when a 64 bit version of task manager is used to create a dump of a 32 bit process. If this is your case, then use the 32 bit version of task manager which can be found in the SysWOW64 folder. This link describes the problem:
http://blogs.msdn.com/b/tess/archive/2010/09/29/capturing-memory-dumps-for-32-bit-processes-on-an-x64-machine.aspx
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(10e0.cd8): Wake debugger - code 80000007 (first/second chance not available)
eax=7c9bc668 ebx=00000000 ecx=00000000 edx=7c9bc080 esi=00000198 edi=00000000
eip=7c95847c esp=0006fc0c ebp=0006fc7c iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
ntdll!KiFastSystemCallRet:
7c95847c c3 ret
0:000> .ecxr
eax=7c9bc668 ebx=00000000 ecx=00000000 edx=7c9bc080 esi=00000198 edi=00000000
eip=7c95847c esp=0006fc0c ebp=0006fc7c iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
ntdll!KiFastSystemCallRet:
7c95847c c3 ret
0:000> kb
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr Args to Child
0006fc08 7c957b99 7c821d1e 00000198 00000000 ntdll!KiFastSystemCallRet
*** ERROR: Symbol file could not be found. Defaulted to export symbols for kernel32.dll -
0006fc0c 7c821d1e 00000198 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
0006fc7c 7c821c8d 00000198 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0x88
*** ERROR: Symbol file could not be found. Defaulted to export symbols for w3dt.dll -
0006fc90 5a604692 00000198 ffffffff 00000000 kernel32!WaitForSingleObject+0x12
*** ERROR: Module load completed but symbols could not be loaded for w3wp.exe
0006ff0c 0100187c 00000007 002e3bd8 00000000 w3dt!IPM_MESSAGE_PIPE::operator=+0x1b31
0006ff44 01001a27 00000007 002e3bd8 002e46d8 w3wp+0x187c
0006ffc0 7c82f23b 00000000 00000000 7ffdf000 w3wp+0x1a27
0006fff0 00000000 010018f8 00000000 78746341 kernel32!ProcessIdToSessionId+0x209
load the symbols...and just wait....use the common command,like the '!threads',
and it works so slowly,here is the kb result,how to solve this problem?
waitting for a long time,and it shows:
0:000> .load sos
0:000> !threadpool
*** ERROR: Symbol file could not be found. Defaulted to export symbols for mscorwks.dll -
PDB symbol for mscorwks.dll not loaded
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of mscorwks.dll is
in the version directory
3) or, if you are debugging a dump file, verify that the file
mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file.
For example, an IA64 dump file must be debugged on an IA64
machine.
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
If you are debugging a minidump, you need to make sure that your executable
path is pointing to mscorwks.dll as well.
so what's the problem?
Is your Internet connection slow?
WinDBG would fetch some symbol images from Mircosoft's site if they are not locally available. It occurs to me once that windbg almost freezes when loading some symbols because of my slow Internet connection.
Well, windbg is slow. period :)
You can try and use local symbols , but if you are using the MS symbol server be ready for some slow debugging.