Picadm is a system driver written by us. We have enabled special pool to make sure the Blue screen happens at the point of corruption.
Blue screen happens with following information:
*
Bugcheck Analysis *
*
BAD_POOL_CALLER (c2)
The current thread is making a bad pool request. Typically this is at a bad IRQL level or double freeing the same allocation, etc.
Arguments:
Arg1: 0000000000000007, Attempt to free pool which was already freed
Arg2: 0000000000001097, (reserved)
Arg3: 0000000000210007, Memory contents of the pool block
Arg4: fffff8a004b98e00, Address of the block of pool being deallocated*
Above information shows that fffff8a004b98e00 is being freed twice leading to BSOD. Since special pool is enabled, we can check the allocations and frees history for this memory address. It gives following result:
*2: kd> !verifier 0x80 fffff8a0`04b98e00
Log of recent kernel pool Allocate and Free operations:
There are up to 0x10000 entries in the log.
Parsing 0x0000000000010000 log entries, searching for address 0xfffff8a004b98e00.
======================================================================
Pool block fffff8a004b98df0, Size 0000000000000210, Thread fffffa80122674f0
fffff80001b0bc9a nt!VfFreePoolNotification+0x4a
fffff800017a367c nt!ExDeferredFreePool+0x126d
fffff8000165b880 nt!MiDeleteSegmentPages+0x35c
fffff8000195cf2f nt!MiSegmentDelete+0x7b
fffff80001637e07 nt!MiCleanSection+0x2f7
fffff80001676754 nt!ObfDereferenceObject+0xd4
fffff80001661170 nt!CcDeleteSharedCacheMap+0x1bc
fffff80001699880 nt!CcUninitializeCacheMap+0x2f0
fffff880030ecfa6 picadm!OwCommonCleanup+0x4b6
fffff880030ec840 picadm!FsdCleanup+0x2a8
fffff880030ec994 picadm!OwFsdCleanup+0x38
fffff80001b16750 nt!IovCallDriver+0xa0
fffff800019824bf nt!IopCloseFile+0x11f
======================================================================
Pool block fffff8a004b98df0, Size 0000000000000210, Thread fffffa80122674f0
fffff80001b0bc9a nt!VfFreePoolNotification+0x4a
fffff800017a367c nt!ExDeferredFreePool+0x126d
fffff8000165b880 nt!MiDeleteSegmentPages+0x35c
fffff8000195cf2f nt!MiSegmentDelete+0x7b
fffff80001637e07 nt!MiCleanSection+0x2f7
fffff80001676754 nt!ObfDereferenceObject+0xd4
fffff80001661170 nt!CcDeleteSharedCacheMap+0x1bc
fffff80001699880 nt!**CcUninitializeCacheMap**+0x2f0
fffff880030ecfa6 picadm!OwCommonCleanup+0x4b6
fffff880030ec840 picadm!FsdCleanup+0x2a8
fffff880030ec994 picadm!OwFsdCleanup+0x38
fffff80001b16750 nt!IovCallDriver+0xa0
fffff800019824bf nt!IopCloseFile+0x11f*
Above shows that this address is deleted twice.
Query : It seems pretty strange to me that both thw stack trace are exactly same. Even the thread is same. What are the possible reasons of this happening. I checked the code involved in the stack trace and cannot find any while/do/for or jump statement leading to execution of the CcUninitializeCacheMap twice.
Below is the thread stack at the time BSOD happened. This is the same thread from where delete has happened:
*2: kd> !thread fffffa80122674f0 7
THREAD fffffa80122674f0 Cid 3de4.41b8 Teb: 000007fffffac000 Win32Thread: fffff900c0ad4850 RUNNING on processor 3
IRP List:
fffffa800a3f1240: (0006,0118) Flags: 00000404 Mdl: 00000000
Not impersonating
DeviceMap fffff8a00fce5bc0
Owning Process fffffa8015ca0060 Image: mstsc.exe
Attached Process N/A Image: N/A
Wait Start TickCount 104601006 Ticks: 0
Context Switch Count 27964 LargeStack
UserTime 00:00:01.375
KernelTime 00:00:07.015
Win32 Start Address 0x000007feef84af90
Stack Init fffff8801406edb0 Current fffff8801406e300
Base fffff8801406f000 Limit fffff88014066000 Call 0
Priority 13 BasePriority 10 UnusualBoost 1 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP RetAddr : Args to Child : Call Site
fffff880`1406e3d0 fffff880`0312a75a : 00ffffff`00100011 00000000`0000002b fffff880`03166f88 fffff880`0313000e : picadm!CDF_TraceRoutine+0x401 [p:\src\ica\hostcore\workstation\picadm\win64\retail\obj\pdm.tmh # 4193]
fffff880`1406e530 fffff880`03146d9e : 00ffffff`00100011 00000000`0000000e fffff880`03166f88 fffff880`031590a0 : picadm!WPP_SF_sq+0xba [p:\src\ica\hostcore\workstation\picadm\win64\retail\obj\directory.tmh # 837]
fffff880`1406e5a0 fffff880`0313674d : fffff980`3a7ecec0 00000000`00000017 fffff880`03166d50 fffff880`031587b0 : picadm!PdmObjectWriteLock+0x6e [p:\src\ica\hostcore\workstation\picadm\pdmobject.cpp # 175]
fffff880`1406e5f0 fffff880`0313e968 : fffff980`3a7ecec0 00000000`00076615 fffff880`03166f30 fffff880`03158b30 : picadm!NodeReleaseShare+0xcd [p:\src\ica\hostcore\workstation\picadm\node.cpp # 529]
fffff880`1406e650 fffff880`03101f56 : 00000000`00076615 00000000`00000001 fffff980`bf03cf50 fffff880`0311f0cd : picadm!PdmFsdRemoveShareAccess+0x128 [p:\src\ica\hostcore\workstation\picadm\pdm.cpp # 3541]
fffff880`1406e6b0 fffff880`030e6943 : 00000000`00000000 fffff980`c3fe4f30 00000000`00000000 00000000`00000001 : picadm!OwRemoveShareAccessFsd+0xfe [h:\devtrees\fsdk_osr\fsdksrc\v1\src\wrapper\fsdsup.cpp # 4729]
fffff880`1406e760 fffff880`030ed373 : 00000000`00000004 00000000`00000000 00000000`00000004 00000000`00000000 : picadm!OwRemoveShareAccess+0xf7 [h:\devtrees\fsdk_osr\fsdksrc\v1\src\wrapper\misc.cpp # 1985]
fffff880`1406e7f0 fffff880`030ec840 : 00000000`00000000 00000000`00000000 fffffa80`15ca0060 fffff880`03175e00 : picadm!OwCommonCleanup+0x883 [h:\devtrees\fsdk_osr\fsdksrc\v1\src\wrapper\cleanup.cpp # 1294]
fffff880`1406e8e0 fffff880`030ec994 : 00000000`b1040100 fffff980`c3fe4f30 fffffa80`057c3e40 fffff880`1406e9e8 : picadm!FsdCleanup+0x2a8 [h:\devtrees\fsdk_osr\fsdksrc\v1\src\wrapper\cleanup.cpp # 255]
fffff880`1406e9a0 fffff800`01b16750 : fffffa80`0a3f1240 00000000`00000000 fffffa80`057c3e01 fffff980`33124fc0 : picadm!OwFsdCleanup+0x38 [h:\devtrees\fsdk_osr\fsdksrc\v1\src\wrapper\cleanup.cpp # 364]
fffff880`1406e9d0 fffff800`019824bf : fffffa80`0a3f1240 fffffa80`15ca0060 00000000`00000000 fffffa80`07419090 : nt!IovCallDriver+0xa0
fffff880`1406ea30 fffff800`01968384 : 00000000`00000000 fffffa80`15ca0060 fffff880`1406eae0 00000000`00000018 : nt!IopCloseFile+0x11f
fffff880`1406eac0 fffff800`01981fb1 : fffffa80`15ca0060 fffffa80`00000001 fffff8a0`11e650d0 00000000`00000000 : nt!ObpDecrementHandleCount+0xb4
fffff880`1406eb40 fffff800`01981ec4 : 00000000`00000570 fffffa80`15ca0060 fffff8a0`11e650d0 00000000`00000570 : nt!ObpCloseHandleTableEntry+0xb1
fffff880`1406ebd0 fffff800`016707d3 : fffffa80`122674f0 fffff880`1406eca0 00000000`063d7240 00000000`00000000 : nt!ObpCloseHandle+0x94
fffff880`1406ec20 00000000`7764f7aa : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame # fffff880`1406ec20)
00000000`0483eab8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x7764f7aa*
Please any help is much appreciated!
Reentrancy can cause this issue. The OS calls your handler more than once on different occasions (I assume it's part of file closing scenario). For example, it's very possible you try to refer to an already closed file handler for the second call - or some other bug of that kind. Check your internal resources management.
Related
I read in Windows Internals that when a thread is created, by default 1 MB of virtual memory is reserved for the user stack. Out of this 1 MB, only the first page (0x1000) will be committed.
I can see this when i dump the image header using dumpbin.exe. Here is what dumpbin shows:
However when i dump the address space of this exe in Windbg using !address command, I see a difference. Windbg shows me that the initial committed size is equal to 3 pages i.e 0x3000
Does anyone know why there is a difference between the initial stack commit size that the image header and debugger shows?
That's a nice question and the key to the answer is understanding what the initial breakpoint is. How initial is it, for starters?
TLDR: The initial breakpoint it too late. That stack has already grown.
You haven't shared the binary you're dealing with, so I chose a binary that exhibits the same behaviour - cacls.exe on 64-bit Windows 10 (file version: 10.0.14393.0).
During the initial breakpoint we observe:
CommandLine: "c:\Windows\System32\cacls.exe"
Symbol search path is: srv*
Executable search path is:
ModLoad: 00007ff6`83bd0000 00007ff6`83bdc000 cacls.exe
ModLoad: 00007ff8`29ce0000 00007ff8`29eb1000 ntdll.dll
ModLoad: 00007ff8`27500000 00007ff8`275ab000 C:\Windows\System32\KERNEL32.DLL
ModLoad: 00007ff8`26f30000 00007ff8`2714d000 C:\Windows\System32\KERNELBASE.dll
ModLoad: 00007ff8`280b0000 00007ff8`2814e000 C:\Windows\System32\msvcrt.dll
ModLoad: 00007ff8`29b10000 00007ff8`29bb2000 C:\Windows\System32\advapi32.dll
ModLoad: 00007ff8`273d0000 00007ff8`27429000 C:\Windows\System32\sechost.dll
ModLoad: 00007ff8`254f0000 00007ff8`25522000 c:\Windows\System32\NTMARTA.dll
ModLoad: 00007ff8`27150000 00007ff8`27245000 C:\Windows\System32\ucrtbase.dll
ModLoad: 00007ff8`277c0000 00007ff8`278e1000 C:\Windows\System32\RPCRT4.dll
(1310.17b0): Break instruction exception - code 80000003 (first chance)
ntdll!LdrpDoDebuggerBreak+0x30:
00007ff8`29db34e0 cc int 3
0:000> !dh -f cacls
File Type: EXECUTABLE IMAGE
FILE HEADER VALUES
8664 machine (X64)
6 number of sections
57899A04 time date stamp Sat Jul 16 05:20:52 2016
0 file pointer to symbol table
0 number of symbols
F0 size of optional header
22 characteristics
Executable
App can handle >2gb addresses
OPTIONAL HEADER VALUES
20B magic #
14.00 linker version
4C00 size of code
3600 size of initialized data
0 size of uninitialized data
52F0 address of entry point
1000 base of code
----- new -----
00007ff683bd0000 image base
1000 section alignment
200 file alignment
3 subsystem (Windows CUI)
10.00 operating system version
10.00 image version
10.00 subsystem version
C000 size of image
400 size of headers
AF10 checksum
0000000000080000 size of stack reserve
0000000000002000 size of stack commit
0000000000100000 size of heap reserve
0000000000001000 size of heap commit
C160 DLL characteristics
High entropy VA supported
Dynamic base
NX compatible
Guard
Terminal server aware
0 [ 0] address [size] of Export Directory
7010 [ 1CC] address [size] of Import Directory
A000 [ 7F0] address [size] of Resource Directory
9000 [ 2DC] address [size] of Exception Directory
0 [ 0] address [size] of Security Directory
B000 [ 54] address [size] of Base Relocation Directory
6A10 [ 38] address [size] of Debug Directory
0 [ 0] address [size] of Description Directory
0 [ 0] address [size] of Special Directory
0 [ 0] address [size] of Thread Storage Directory
60E0 [ D0] address [size] of Load Configuration Directory
0 [ 0] address [size] of Bound Import Directory
61B0 [ 3B8] address [size] of Import Address Table Directory
0 [ 0] address [size] of Delay Import Directory
0 [ 0] address [size] of COR20 Header Directory
0 [ 0] address [size] of Reserved Directory
0:000> !address #rsp
Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...
Usage: Stack
Base Address: 00000049`8fbbc000
End Address: 00000049`8fbc0000
Region Size: 00000000`00004000 ( 16.000 kB)
State: 00001000 MEM_COMMIT
Protect: 00000004 PAGE_READWRITE
Type: 00020000 MEM_PRIVATE
Allocation Base: 00000049`8fb40000
Allocation Protect: 00000004 PAGE_READWRITE
More info: ~0k
Content source: 1 (target), length: 180
We see the initial stack commit size is 0x2000, but 0x4000 is actually committed.
But that's already very late during the process initialization process (no pun intended). All the import DLL are already loaded, for example.
The so-called "initial break-point" is simply a (more or less1) hardcoded int 3 instruction called by the process initializtion code in NTDLL. If you look at the stack at this point you'll see the aptly named LdrpDoDebuggerBreak function which is called by LdrpInitializeProcess:
0:000> k
# Child-SP RetAddr Call Site
00 00000049`8fbbee80 00007ff8`29d72e22 ntdll!LdrpDoDebuggerBreak+0x30
01 00000049`8fbbeec0 00007ff8`29da8986 ntdll!LdrpInitializeProcess+0x1962
02 00000049`8fbbf2c0 00007ff8`29d59fae ntdll!_LdrpInitialize+0x4e982
03 00000049`8fbbf340 00000000`00000000 ntdll!LdrInitializeThunk+0xe
By the time that happened the stack has already been used (for example, to load statically linked DLLs and perform their initialization code), so it shouldn't be much surprise that the stack has grown already.
To examine the process when it has just been created we need to break on the process creation event rather than on the initial breakpoint (which isn't that initial as we now understand).
We can do that either using sxe cpr and .restarting like I did or running WinDbg/NTSD with -xe cpr. Doing that reveals something interesting2:
0:000> .restart
CommandLine: C:\Windows\System32\cacls.exe
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred srv*
Symbol search path is: srv*
Executable search path is:
ModLoad: 00007ff6`83bd0000 00007ff6`83bdc000 cacls.exe
00007ff8`29d470b0 4883ec48 sub rsp,48h
0:000> .imgscan /l
MZ at 00007ff6`83bd0000, prot 00000002, type 01000000 - size c000
Name: cacls.exe
Loaded cacls.exe module
MZ at 00007ff8`29ce0000, prot 00000002, type 01000000 - size 1d1000
Name: ntdll.dll
Loaded ntdll.dll module
0:000> !address #rsp
Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...
Usage: Stack
Base Address: 0000004a`5428e000
End Address: 0000004a`54290000
Region Size: 00000000`00002000 ( 8.000 kB)
State: 00001000 MEM_COMMIT
Protect: 00000004 PAGE_READWRITE
Type: 00020000 MEM_PRIVATE
Allocation Base: 0000004a`54210000
Allocation Protect: 00000004 PAGE_READWRITE
More info: ~0k
Content source: 1 (target), length: 478
The committed region size is 0x2000 - like the header says!
If we let it continue we'll eventually get to the initial breakpoint with more stack comitted.
1I said more or less hardcoded because the actual code of the function is
0:000> uf ntdll!LdrpDoDebuggerBreak
ntdll!LdrpDoDebuggerBreak:
00007ff8`29db34b0 4883ec38 sub rsp,38h
00007ff8`29db34b4 488364242000 and qword ptr [rsp+20h],0
00007ff8`29db34ba 41b901000000 mov r9d,1
00007ff8`29db34c0 4c8d442440 lea r8,[rsp+40h]
00007ff8`29db34c5 418d5110 lea edx,[r9+10h]
00007ff8`29db34c9 48c7c1feffffff mov rcx,0FFFFFFFFFFFFFFFEh
00007ff8`29db34d0 e88b30fdff call ntdll!NtQueryInformationThread (00007ff8`29d86560)
00007ff8`29db34d5 85c0 test eax,eax
00007ff8`29db34d7 780a js ntdll!LdrpDoDebuggerBreak+0x33 (00007ff8`29db34e3) Branch
ntdll!LdrpDoDebuggerBreak+0x29:
00007ff8`29db34d9 807c244000 cmp byte ptr [rsp+40h],0
00007ff8`29db34de 7503 jne ntdll!LdrpDoDebuggerBreak+0x33 (00007ff8`29db34e3) Branch
ntdll!LdrpDoDebuggerBreak+0x30:
00007ff8`29db34e0 cc int 3
00007ff8`29db34e1 eb00 jmp ntdll!LdrpDoDebuggerBreak+0x33 (00007ff8`29db34e3) Branch
ntdll!LdrpDoDebuggerBreak+0x33:
00007ff8`29db34e3 4883c438 add rsp,38h
00007ff8`29db34e7 c3 ret
It does stuff like checking whether or not this thread is to be "hidden from the debugger", but basically it just breaks into the debugger.
2The .imgscan /l is needed because without it we get:
0:000> !address #rsp
No symbols for ntdll. Cannot continue.
I started using WinDbg to debug VB6 runtime errors but can't really get it to work, all I need is to know the source of the error (the procedure where the error comes from).
I created a small app to test and made it to throw an overflow runtime error like this:
Private Sub Command1_Click()
Dim a As Byte
a = 1000
End Sub
I compiled it with "Create Symbolic debug Info" option checked to create the PDB file.
Then I attached the application to WinDbg and clicked the button to throw the error, but when I check the call stack I don't find any trace of the Command1_Click procedure. All I get is the following :
0:001> ~* k
0 Id: 56c.173c Suspend: 1 Teb: 7ffde000 Unfrozen
ChildEBP RetAddr
0012ea08 7e419418 ntdll!KiFastSystemCallRet
0012ea40 7e4249c4 USER32!NtUserWaitMessage+0xc
0012ea68 7e43a956 USER32!InternalDialogBox+0xd0
0012ed28 7e43a2bc USER32!SoftModalMessageBox+0x938
0012ee78 7e43a10b USER32!MessageBoxWorker+0x2ba
0012eee4 729af829 USER32!MessageBoxIndirectA+0xb8
WARNING: Stack unwind information not available. Following frames may be wrong.
0012ef24 729af6a5 MSVBVM60!IID_IVbaHost+0x411e9
0012ef4c 729af9a0 MSVBVM60!IID_IVbaHost+0x41065
0012ef7c 729a3d68 MSVBVM60!IID_IVbaHost+0x41360
0012efe0 729a3db6 MSVBVM60!IID_IVbaHost+0x35728
0012f000 72a0c411 MSVBVM60!IID_IVbaHost+0x35776
0012f01c 72a0c6f3 MSVBVM60!_vbaOnGoCheck+0xba
0012f05c 7c9032a8 MSVBVM60!EbGetErrorInfo+0x115
0012f080 7c90327a ntdll!ExecuteHandler2+0x26
0012f130 7c90e46a ntdll!ExecuteHandler+0x24
0012f130 7c812aeb ntdll!KiUserExceptionDispatcher+0xe
0012f484 72a10dcf kernel32!RaiseException+0x53
0012f4a4 72a0e228 MSVBVM60!EbGetHandleOfExecutingProject+0x22b3
0012f4b8 72a0e28c MSVBVM60!rtcDoEvents+0x131
0012f4c8 72a219ee MSVBVM60!rtcDoEvents+0x195
0012f644 72992667 MSVBVM60!_vbaUI1I2+0x12
0012f668 729f4657 MSVBVM60!IID_IVbaHost+0x24027
0012f698 7299ce49 MSVBVM60!DllCanUnloadNow+0x149a5
0012f6c0 7299f97d MSVBVM60!IID_IVbaHost+0x2e809
0012f71c 7299e22c MSVBVM60!IID_IVbaHost+0x3133d
0012f740 7299dc6d MSVBVM60!IID_IVbaHost+0x2fbec
0012f7ac 729c223a MSVBVM60!IID_IVbaHost+0x2f62d
0012f92c 7299ce49 MSVBVM60!IID_IVbaHost+0x53bfa
0012f954 7299f97d MSVBVM60!IID_IVbaHost+0x2e809
0012f9b0 7e418734 MSVBVM60!IID_IVbaHost+0x3133d
0012f9dc 7e418816 USER32!InternalCallWinProc+0x28
0012fa44 7e42927b USER32!UserCallWinProcCheckWow+0x150
0012fa80 7e4292e3 USER32!SendMessageWorker+0x4a5
0012faa0 7e44ff7d USER32!SendMessageW+0x7f
0012fab8 7e4465d2 USER32!xxxButtonNotifyParent+0x41
0012fad4 7e425e94 USER32!xxxBNReleaseCapture+0xf8
0012fb58 7e43b082 USER32!ButtonWndProcWorker+0x6df
0012fb78 7e418734 USER32!ButtonWndProcA+0x5d
0012fba4 7e418816 USER32!InternalCallWinProc+0x28
0012fc0c 7e42a013 USER32!UserCallWinProcCheckWow+0x150
0012fc3c 7e42a998 USER32!CallWindowProcAorW+0x98
0012fc5c 7299d082 USER32!CallWindowProcA+0x1b
0012fcc8 729f492d MSVBVM60!IID_IVbaHost+0x2ea42
0012fcf0 7299ce49 MSVBVM60!DllCanUnloadNow+0x14c7b
0012fd18 7299f97d MSVBVM60!IID_IVbaHost+0x2e809
0012fd74 7e418734 MSVBVM60!IID_IVbaHost+0x3133d
0012fda0 7e418816 USER32!InternalCallWinProc+0x28
0012fe08 7e4189cd USER32!UserCallWinProcCheckWow+0x150
0012fe68 7e4196c7 USER32!DispatchMessageWorker+0x306
0012fe78 7294a6c8 USER32!DispatchMessageA+0xf
0012feb8 7294a63f MSVBVM60!_vbaStrToAnsi+0x2f1
0012fefc 7294a51d MSVBVM60!_vbaStrToAnsi+0x268
0012ff18 7294a4e8 MSVBVM60!_vbaStrToAnsi+0x146
0012ff3c 72943644 MSVBVM60!_vbaStrToAnsi+0x111
0012ffb8 00401246 MSVBVM60!ThunRTMain+0xa0
0012fff0 00000000 Project1!__vbaS+0xa
# 1 Id: 56c.10a8 Suspend: 1 Teb: 7ffdd000 Unfrozen
ChildEBP RetAddr
00f0ffc8 7c950010 ntdll!DbgBreakPoint
00f0fff4 00000000 ntdll!DbgUiRemoteBreakin+0x2d
Note that WinDbg doesn't break on the error, so I manually break into it after I get the error message and before clicking OK, because if I clicked OK, the application closes.
I'm new to WinDbg but I read that it should break on error but it doesn't, all I get is "First chance Exception" and I don't get a "Second chance Exception" when I click OK and let it crash.
I think that somehow WinDbg considers that VB6 handled the error even though it didn't handle it. Is there something I'm missing here? How can I get the trace of the error to the calling procedure?
Running the application in VB6 itself results in a VB6 runtime error 6, which stands for "overflow".
First chance exception
Running the application under WinDbg (6.2.9200), I first see a
(6cc.6d4): Unknown exception - code c000008f (first chance)
which is probably the most common VB6 exception. By default, WinDbg does not break on first chance exceptions of this type. If you want it to do so, you need to enable that explicitly by
sxe c000008f
Second chance exception
The first chance exception is followed by a message box
This message box is the first indicator that the exception must have been caught, otherwise there would have been a second chance exception immediately afterwards.
And in fact, VB6 creates an exception handler:
0:000> u
Project1!Form1::Command_Click [Form1 # 5]:
004019a0 55 push ebp
004019a1 8bec mov ebp,esp
004019a3 83ec0c sub esp,0Ch
004019a6 68b6104000 push offset Project1!__vbaExceptHandler (004010b6)
004019ab 64a100000000 mov eax,dword ptr fs:[00000000h]
...
004019df b9e8030000 mov ecx,3E8h
004019e4 ff1530104000 call dword ptr [Project1!_imp___vbaUI1I2 (00401030)]
...
At 004019a6 you see that VB6 is working with __vbaExceptHandler. mov dword ptr fs:[0],esp is the beginning of a try/catch block. At 004019df it stores 0x3E8 (hex) or 1000 (dec) to ECX. That's where your code is.
After confirming the message, the process terminates, but not due to a second chance exception but because of an ExitProcess() call. That's the way VB6 "handles" exceptions.
0:000> k
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
0012ff28 7c81bfb6 ntdll!KiFastSystemCallRet
0012ff3c 73393657 kernel32!ExitProcess+0x14
0012ffb8 0040113a MSVBVM60!ThunRTMain+0xb3
0012fff0 00000000 image00400000+0x113a
Symbols
To see if the symbols have been loaded correctly, type lm (list modules). It should look like this:
0:000> lm
start end module name
00400000 00404000 Project1 (private pdb symbols) C:\Programme\Microsoft Visual Studio\VB98\Project1.pdb
If you see (deferred), the symbols have not been loaded yet. Type ld Project1 to load them.
With symbols, you can set breakpoints by first finding the method with x and then setting a breakpoint bp:
0:000> x Project1!Form1*
004019a0 Project1!Form1::Command_Click (void)
0:000> bp Project1!Form1::Command_Click
0:000> bl
0 e 004019a0 0001 (0001) 0:**** Project1!Form1::Command_Click
[...]
Breakpoint 0 hit
eax=004016b4 ebx=00000001 ecx=00000000 edx=733a3dd8 esi=0012f5e4 edi=0012f514
eip=004019a0 esp=0012f50c ebp=0012f514 iopl=0 nv up ei pl nz ac pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000216
Project1!Form1::Command_Click:
004019a0 55 push ebp
0:000> k
ChildEBP RetAddr
0012f508 733e1ce3 Project1!Form1::Command_Click [Form1 # 5]
WARNING: Stack unwind information not available. Following frames may be wrong.
0012f524 733e1fe4 MSVBVM60!IID_IVbaHost+0x23703
...
But a breakpoint is not what you wanted...
Why the method is not on the call stack
Unfortunately enabling first chance exceptions and loading symbols does still not help seeing the method on the call stack at the time the exception is thrown. It's somewhere in DoEvents():
0:000> .exr -1
ExceptionAddress: 7c812fd3 (kernel32!RaiseException+0x00000052)
ExceptionCode: c000008f
ExceptionFlags: 00000001
NumberParameters: 2
Parameter[0]: deadcafe
Parameter[1]: deadcafe
0:000> k
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
0012f484 73460c29 kernel32!RaiseException+0x52
0012f4a4 7345e082 MSVBVM60!EbGetHandleOfExecutingProject+0x22b3
0012f4b8 7345e0e6 MSVBVM60!rtcDoEvents+0x131
...
Have you noticed the warning about the stack unwind information? Let's look at the stack manually:
0:000> dps #ebp
0012f484 0012f4a4
0012f488 73460c29 MSVBVM60!EbGetHandleOfExecutingProject+0x22b3
0012f48c c000008f
0012f490 00000001
0012f494 00000002
0012f498 0012f49c
0012f49c deadcafe
0012f4a0 deadcafe
...
0012f4d8 004019e7 Project1!Form1::Command_Click+0x53 [Form1 # 7]
0012f4dc 0012f514
0012f4e0 0012f5e4
0012f4e4 00000001
0012f4e8 00000000
0012f4ec 00000000
0012f4f0 00000000
0012f4f4 0012fa34
0012f4f8 004010b6 Project1!__vbaExceptHandler
I guess you can't do anything about it. VB6 is not very good in creating symbols and also not in maintaining a nice call stack.
I'm currently trying to debug a system deadlock and I'm having a hard time understanding this.
Child-SP RetAddr : Args to Child : Call Site
fffff880`035cb760 fffff800`02ecef72 : 00000000`00000002 fffffa80`066e8b50 00000000`00000000 fffffa80`066a16e0 : nt!KiSwapContext+0x7a
fffff880`035cb8a0 fffff800`02ee039f : fffffa80`0b9256b0 00000000`000007ff 00000000`00000000 00000000`00000000 : nt!KiCommitThreadWait+0x1d2
fffff880`035cb930 fffff880`0312a5e4 : 00000000`00000000 fffff800`00000000 fffffa80`079a3c00 00000000`00000000 : nt!KeWaitForSingleObject+0x19
Why would the first argument for KeWaitForSingleObject be null?
Unless I'm misunderstanding isn't the first argument the object being waited on?
Is the deadlock simply that this thread is waiting on nothing or is this ordinary behavior?
Additionally I see another process (services.exe) showing a similar stack trace:
1: kd> .thread fffffa800d406b50
Implicit thread is now fffffa80`0d406b50
1: kd> kv
*** Stack trace for last set context - .thread/.cxr resets it
Child-SP RetAddr : Args to Child : Call Site
fffff880`09ed4800 fffff800`02ecef72 : fffffa80`0d406b50 fffffa80`0d406b50 00000000`00000000 fffff8a0`00000000 : nt!KiSwapContext+0x7a
fffff880`09ed4940 fffff800`02ee039f : 00000000`000000b4 fffffa80`0b1df7f0 00000000`0000005e fffff800`031ae5e7 : nt!KiCommitThreadWait+0x1d2
fffff880`09ed49d0 fffff800`031d1e3e : fffffa80`0d406b00 00000000`00000006 00000000`00000001 00000000`093bf000 : nt!KeWaitForSingleObject+0x19f
fffff880`09ed4a70 fffff800`02ed87d3 : fffffa80`0d406b50 00000000`77502410 fffff880`09ed4ab8 fffffa80`0b171a50 : nt!NtWaitForSingleObject+0xde
Is this thread waiting on itself essentially?
You're debugging a 64-bit process.
Remember the x64 calling convention, which is explained here. The first 4 arguments are passed in registers. After that, arguments are pushed onto the stack.
Unfortunately, kv blindly displays the stack arguments. In fact, it would be quite difficult (and sometimes impossible) for it to determine what the first 4 arguments actually were at the time of the call since they may not have been stored anywhere that can ever be recovered.
So, you are looking at the 5th argument to nt!NtWaitForSingleObject, where a nullptr is a pretty typical argument for a Timeout.
Luckily for us debugging types, all is not lost! There is a windbg extension which does its best to reconstruct the arguments when the function was called. The extension is called CMKD. You can place the extension DLL in your winext folder and call it like so:
0:000> !cmkd.stack -p
Call Stack : 7 frames
## Stack-Pointer Return-Address Call-Site
00 000000a408c7fb28 00007ffda95b1148 ntdll!NtWaitForSingleObject+a
Parameter[0] = 0000000000000034
Parameter[1] = 0000000000000000
Parameter[2] = 0000000000000000
Parameter[3] = (unknown)
01 000000a408c7fb30 00007ff7e44c13f1 KERNELBASE!WaitForSingleObjectEx+98
Parameter[0] = 0000000000000034
Parameter[1] = 00000000ffffffff
Parameter[2] = 0000000000000000
Parameter[3] = 00007ff7e44cba28
02 000000a408c7fbd0 00007ff7e44c3fed ConsoleApplication2!main+41
Parameter[0] = (unknown)
Parameter[1] = (unknown)
Parameter[2] = (unknown)
Parameter[3] = (unknown)
Notice that it does not always succeed at finding the argument, as some of them are (unknown). But, it does a pretty good job and can be an invaluable tool when debugging 64-bit code.
This looks like a 64-bit OS, and therefore the calling convention is not to pass all the parameters on the stack. Rather, the first four parameters get passed in RCX, RDX, R8, and R9, with the remaining parameters on the stack. So if you catch the call to KeWaitForSingleObject it's easy to see what's in RCX and go from there. Once you are a few stack frames beyond that, it's much hard to tell since something will have been loaded into that register. The original value is probably stored somewhere, but it will be difficult to find.
The gwan (4.1.18) instance runs for about one day, and then crashed with this tracing information (Not suer how useful it is:( How to get meaningful tracing information?)
----------------
----------------
Sun, 27 Jan 2013 20:07:34 GMT: warning: no loadable sections found in added symbol-file /home/gwan/gwan
0x00007f0018206148 in ?? ()
Id Target Id Frame
* 1 process 2860 "gwan" 0x00007f0018206148 in ?? ()
Thread 1 (process 2860):
#0 0x00007f0018206148 in ?? ()
#1 0x0000000000000004 in ?? ()
#2 0x00007f0018941245 in ?? ()
#3 0x00007f0018206070 in ?? ()
#4 0x00007f000b70ece8 in ?? ()
#5 0x000000000000002e in ?? ()
#6 0x0000000000000000 in ?? ()
Signal : 11:Address not mapped to object
Signal src : 1:.
errno : 0
Thread : 1
Code Pointer: 0000004343c0 (module:gwan, function:??, line:0)
Access Address: 0001004647f0
Registers : EAX=7f000ca04d40 CS=00000033 EIP=0000004343c0 EFLGS=000000010282
EBX=7f000ca08eb0 SS=00000000 ESP=7f000ca04af0 EBP=00000001aeb0
ECX=0000ffffffb0 DS=00000000 ESI=7f000ca04cf0 FS=00000033
EDX=0000000000b0 ES=00000000 EDI=7f000ca04d9c CS=00000033
Module :Function :Line # PgrmCntr(EIP) RetAddress FramePtr(EBP)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
How to get meaningful tracing information?
Leaving the relevant per-thread crash information would surely have helped to give you some insights.
But as you have removed it by hand, I am assuming that you know what you are doing.
Note that running gwan under /home/gwan/gwan -d is not a good idea.
Rather use something like /opt/gwan/gwan -d because the contents of /home are not reachable from other accounts.
And if you are running during days without the parent angel process (-d mode) then it's even less a good idea.
That's not G-WAN-specific:
"you should NEVER run Nginx in production with 'master_process off'"
I am trying to see the IMAGE_DOS_HEADER structure using the 'dt' command in windbg:
0:001> dt ntdll!IMAGE_DOS_HEADER
**Symbol ntdll!IMAGE_DOS_HEADER not found**
I searched and found that others have been able to do this:
MSDN Blogs > Junyoung's Blog > Portable Executable File Format on Memory Dump
0:000> dt IMAGE_DOS_HEADER 77c00000
ntdll!IMAGE_DOS_HEADER
+0x000 e_magic : 0x5a4d ? MZ
+0x002 e_cblp : 0x90
+0x004 e_cp : 3
+0x006 e_crlc : 0
+0x008 e_cparhdr : 4
+0x00a e_minalloc : 0
+0x00c e_maxalloc : 0xffff
+0x00e e_ss : 0
+0x010 e_sp : 0xb8
+0x012 e_csum : 0
+0x014 e_ip : 0
+0x016 e_cs : 0
+0x018 e_lfarlc : 0x40
+0x01a e_ovno : 0
+0x01c e_res : [4] 0
+0x024 e_oemid : 0
+0x026 e_oeminfo : 0
+0x028 e_res2 : [10] 0
+0x03c e_lfanew : 232
Is it possible to add the missing symbol to public symbols of ntdll so that I can access the data structure fields in a windbg debugging session?
This is really weird - I tried the following and got different results on a win7 and on winxp systems -
WinXP:
0:015> dt ntdll!*HEADER*
ntdll!_IMAGE_NT_HEADERS
ntdll!_IMAGE_FILE_HEADER
ntdll!_IMAGE_OPTIONAL_HEADER
ntdll!_SLIST_HEADER
ntdll!_DISPATCHER_HEADER
ntdll!_IMAGE_SECTION_HEADER
Win7:
0:000> dt ntdll!*HEADER*
ntdll!_IMAGE_NT_HEADERS
ntdll!_IMAGE_FILE_HEADER
ntdll!_IMAGE_OPTIONAL_HEADER
ntdll!_IMAGE_DOS_HEADER
ntdll!_SLIST_HEADER
ntdll!_DISPATCHER_HEADER
ntdll!_MM_PAGE_ACCESS_INFO_HEADER
ntdll!_WHEA_ERROR_RECORD_HEADER
ntdll!_HEAP_USERDATA_HEADER
ntdll!_HEAP_USERDATA_HEADER
ntdll!_WHEA_ERROR_RECORD_HEADER_VALIDBITS
ntdll!_WHEA_ERROR_RECORD_HEADER_FLAGS
ntdll!_XSAVE_AREA_HEADER
So it seems that the _IMAGE_DOS_HEADER symbol is stripped from the public symbols of winxp's ntdll.dll - my question remains - how can I add the symbol if I know the structure to a public symbols dll of a dll I don't have the sources for?
This happens, it can be quite frustrating actually (though they've done a good job cleaning a lot of these up).
You can add types to existing PDBs, though it requires that you have the Visual C compiler available (either from Visual Studio or the WDK). I outline the steps in my response here:
http://www.osronline.com/showthread.cfm?link=193747
We're using the kernel there, though you have the same steps. The differences will be:
1) Look for ntdll instead of nt
2) You'll need to write a C file with the definition of the structure that you want.
-scott
you can also check this thread it shows how to add types to
http://www.woodmann.com/forum/showthread.php?10295-Mysteries-of-win32k-amp-GDI&p=72632&viewfull=1#post72632
post # 21
They're not missing, you just don't have your symbol path configured.
Check the following Microsoft KB, if that doesn't work, you may have an odder problem.
One way to make sure you're debugging with symbols is the lm command.
The output without debug symbols is likely to look like this:
lm
start end module name
01000000 01014000 notepad (no symbols)
74720000 7476b000 MSCTF (export symbols) C:\WINDOWS\system32\MSCTF.dl
Whilee the output for lm with debug symbols will be:
lm
start end module name
01000000 01014000 notepad (pdb symbols) e:\LocalSymbols\notepad.pdb\15800B8231AF4FDE85232D42B267D3E51\notepad.pdb