I'm trying to use IOCP to do async file reads.
This works fine 99% of the time.. but that 1% of the time a request seems to never finish and so the operation just waits forever.
The sequence of API calls for the file that hangs is
# Time of Day Relative Time Thread Module Category API Return Value Error Duration
1160 4:14:50.142 PM 0:00:00:308 1 ghc-stage2.exe File Management CreateFileW ( "\\?\<path>\llvm-targets", GENERIC_READ, FILE_SHARE_DELETE | FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, FILE_FLAG_OVERLAPPED | FILE_FLAG_SEQUENTIAL_SCAN, NULL ) 0x00000000000001f0 0.0001230
1173 4:14:50.143 PM 0:00:00:309 1 ghc-stage2.exe File Management CreateIoCompletionPort ( 0x00000000000001f0, 0x00000000000001e4, 496, 0 ) 0x00000000000001e4 0.0000407
1179 4:14:50.143 PM 0:00:00:309 1 ghc-stage2.exe File Management GetFileType ( 0x00000000000001f0 ) FILE_TYPE_DISK 0.0000039
1190 4:14:50.143 PM 0:00:00:309 1 ghc-stage2.exe File Management SetFileCompletionNotificationModes ( 0x00000000000001f0, 3 ) TRUE 0.0000075
1224 4:14:50.144 PM 0:00:00:310 1 ghc-stage2.exe Error Handling GetLastError ( ) ERROR_IO_PENDING 0.0000009
1223 4:14:50.143 PM 0:00:00:309 1 ghc-stage2.exe File Management ReadFile ( 0x00000000000001f0, 0x00000000074d7010, 8192, NULL, 0x00000000073670e0 ) FALSE 997 = Overlapped I/O operation is in progress. 0.0000811
1276 4:14:50.144 PM 0:00:00:310 5 ghc-stage2.exe File Management GetQueuedCompletionStatusEx ( 0x00000000000001e4, 0x0000000007406340, 64, 0x0000000007367140, INFINITE, FALSE )
Even when reading files one at a time this seems to happen and I'm not sure why. I initially thought that the pointer containing the LPOVERLAPPED_ENTRY structure may be becoming invalid, but checking the location with gdb shows it's still good.
(gdb) x/12w 0x0000000007406340
0x7406340: 0x00000000 0x00000000 0x00000000 0x00000000
0x7406350: 0x00000000 0x00000000 0x00000000 0x00000000
0x7406360: 0x00000000 0x00000000 0x00000000 0x00000000
Does anyone have any idea what might be going on?
Related
Using WinDbg Preview or WinDbg from Windows 10 SDK, when launching 32-bit process on Windows 10 1909 (build 18363.815) setting a breakpoint on kernelbase!RegOpenKeyExW by name doesn't work.
Example:
Launch C:\windows\syswow64\notepad.exe under WinDbg
.symfix C:\symbols
.reload
bp ntdll!NtOpenKeyEx
g
k
# ChildEBP RetAddr
00 0308f314 74bb5030 ntdll!NtOpenKeyEx
01 0308f3c4 74bb4b87 KERNELBASE!LocalBaseRegOpenKey+0x110
02 0308f42c 74bb4a3c KERNELBASE!RegOpenKeyExInternalW+0x137
03 0308f450 761c34b9 KERNELBASE!RegOpenKeyExW+0x1c
04 0308f488 761c345c combase!ComVerifierSettings::ReadBooleanFromOleKey+0x35 [onecore\com\combase\verifier\verify.cxx # 1046]
05 0308f4a4 76115745 combase!ComVerifierSettings::ComVerifierSettings+0x2f [onecore\com\combase\verifier\verify.cxx # 768]
06 0308f4a8 756a6cd7 combase!`dynamic initializer for 'ComVerifierSettings::s_singleton''+0x5 [onecore\com\combase\verifier\verify.cxx # 626]
07 0308f4c0 761e1801 ucrtbase!_initterm+0x37
08 0308f500 761e175d combase!dllmain_crt_process_attach+0x8c [vccrt\vcstartup\src\startup\dll_dllmain.cpp # 64]
09 0308f510 761e196a combase!dllmain_crt_dispatch+0x3d [vccrt\vcstartup\src\startup\dll_dllmain.cpp # 138]
0a 0308f550 761e1a6e combase!dllmain_dispatch+0x59 [vccrt\vcstartup\src\startup\dll_dllmain.cpp # 195]
0b 0308f564 77071de6 combase!_DllMainCRTStartup+0x1e [vccrt\vcstartup\src\startup\dll_dllmain.cpp # 253]
0c 0308f584 77035608 ntdll!LdrxCallInitRoutine+0x16
0d 0308f5d0 77043f8f ntdll!LdrpCallInitRoutine+0x51
0e 0308f658 77044836 ntdll!LdrpInitializeNode+0x133
0f 0308f67c 7704484d ntdll!LdrpInitializeGraphRecurse+0x5d
10 0308f6a4 770a9542 ntdll!LdrpInitializeGraphRecurse+0x74
11 0308f6b4 770a9382 ntdll!LdrpInitializeGraph+0x13
12 0308f914 77051dd1 ntdll!LdrpInitializeProcess+0x1cc2
13 0308f96c 77051cc1 ntdll!_LdrpInitialize+0xba
14 0308f978 00000000 ntdll!LdrInitializeThunk+0x11
.restart
bc *
bp KERNELBASE!RegOpenKeyExW
g
No breakpoint hit
.restart
bc *
x kernelbase!RegOpenKeyExW*
74bc64b0 KERNELBASE!RegOpenKeyExW (void)
74bb4a20 KERNELBASE!RegOpenKeyExW (_RegOpenKeyExW#20)
uf 74bc64b0
KERNELBASE!EventAccessControl:
74bc64b0 6a7f push 7Fh
74bc64b2 58 pop eax
74bc64b3 c21400 ret 14h
uf 74bb4a20
KERNELBASE!RegOpenKeyExW:
74bb4a20 8bff mov edi,edi
74bb4a22 55 push ebp
74bb4a23 8bec mov ebp,esp
74bb4a25 51 push ecx
74bb4a26 6a00 push 0
74bb4a28 ff7518 push dword ptr [ebp+18h]
74bb4a2b ff7514 push dword ptr [ebp+14h]
74bb4a2e ff7510 push dword ptr [ebp+10h]
74bb4a31 ff750c push dword ptr [ebp+0Ch]
74bb4a34 ff7508 push dword ptr [ebp+8]
74bb4a37 e814000000 call KERNELBASE!RegOpenKeyExInternalW (74bb4a50)
74bb4a3c 59 pop ecx
74bb4a3d 5d pop ebp
74bb4a3e c21400 ret 14h
bp 74bb4a20
g
Breakpoint hit - why do I have to use this address not name? Why the duplicate names?
Breakpoint 0 hit
eax=0308f478 ebx=00000000 ecx=760de820 edx=00000000 esi=760de820 edi=760c8d98
eip=74bb4a20 esp=0308f454 ebp=0308f488 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
KERNELBASE!RegOpenKeyExW:
74bb4a20 8bff mov edi,edi
Checking import table of process, the address referenced is 74bb4a20.
0:000> lmvm notepad
Browse full module list
start end module name
00fe0000 0100b000 notepad (deferred)
Image path: notepad.exe
Image name: notepad.exe
Browse all global symbols functions data
Image was built with /Brepro flag.
Timestamp: 93B4E8FA (This is a reproducible build file hash, not a timestamp)
CheckSum: 00032822
ImageSize: 0002B000
File version: 10.0.18362.693
Product version: 10.0.18362.693
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 1.0 App
File date: 00000000.00000000
Translations: 0409.04b0
Information from resource tables:
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: Notepad
OriginalFilename: NOTEPAD.EXE
ProductVersion: 10.0.18362.693
FileVersion: 10.0.18362.693 (WinBuild.160101.0800)
FileDescription: Notepad
LegalCopyright: © Microsoft Corporation. All rights reserved.
0:000> !dh 00fe0000
File Type: EXECUTABLE IMAGE
FILE HEADER VALUES
14C machine (i386)
6 number of sections
93B4E8FA time date stamp
0 file pointer to symbol table
0 number of symbols
E0 size of optional header
102 characteristics
Executable
32 bit word machine
OPTIONAL HEADER VALUES
10B magic #
14.15 linker version
1FC00 size of code
7400 size of initialized data
0 size of uninitialized data
1F8D0 address of entry point
1000 base of code
----- new -----
00fe0000 image base
1000 section alignment
200 file alignment
2 subsystem (Windows GUI)
10.00 operating system version
10.00 image version
10.00 subsystem version
2B000 size of image
400 size of headers
32822 checksum
00040000 size of stack reserve
00011000 size of stack commit
00100000 size of heap reserve
00001000 size of heap commit
C140 DLL characteristics
Dynamic base
NX compatible
Guard
Terminal server aware
0 [ 0] address [size] of Export Directory
234B8 [ 370] address [size] of Import Directory
27000 [ BE0] address [size] of Resource Directory
0 [ 0] address [size] of Exception Directory
0 [ 0] address [size] of Security Directory
28000 [ 21A8] address [size] of Base Relocation Directory
4A80 [ 54] address [size] of Debug Directory
0 [ 0] address [size] of Description Directory
0 [ 0] address [size] of Special Directory
13D4 [ 18] address [size] of Thread Storage Directory
1330 [ A4] address [size] of Load Configuration Directory
0 [ 0] address [size] of Bound Import Directory
23000 [ 4B4] address [size] of Import Address Table Directory
207A4 [ E0] address [size] of Delay Import Directory
0 [ 0] address [size] of COR20 Header Directory
0 [ 0] address [size] of Reserved Directory
SECTION HEADER #1
.text name
1FB50 virtual size
1000 virtual address
1FC00 size of raw data
400 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
60000020 flags
Code
(no align specified)
Execute Read
Debug Directories(3)
Type Size Address Pointer
cv 24 5078 4478 Format: RSDS, guid, 1, notepad.pdb
( 13) 3a4 509c 449c
( 16) 24 5440 4840
SECTION HEADER #2
.data name
1DB0 virtual size
21000 virtual address
800 size of raw data
20000 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
C0000040 flags
Initialized Data
(no align specified)
Read Write
SECTION HEADER #3
.idata name
2472 virtual size
23000 virtual address
2600 size of raw data
20800 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
40000040 flags
Initialized Data
(no align specified)
Read Only
SECTION HEADER #4
.didat name
78 virtual size
26000 virtual address
200 size of raw data
22E00 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
C0000040 flags
Initialized Data
(no align specified)
Read Write
SECTION HEADER #5
.rsrc name
BE0 virtual size
27000 virtual address
C00 size of raw data
23000 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
40000040 flags
Initialized Data
(no align specified)
Read Only
SECTION HEADER #6
.reloc name
21A8 virtual size
28000 virtual address
2200 size of raw data
23C00 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
42000040 flags
Initialized Data
Discardable
(no align specified)
Read Only
0:000> dps 00fe0000+23000 00fe0000+23000+4B4
01003000 639b5690 COMCTL32!CreateStatusWindowW
01003004 63a48240 COMCTL32!TaskDialogIndirect
01003008 00000000
0100300c 76f04680 GDI32!SelectObject
01003010 76f07930 GDI32!GetTextFaceW
01003014 76f077a0 GDI32!EnumFontsW
01003018 76f079b0 GDI32!TextOutW
0100301c 76f06a80 GDI32!GetTextExtentPoint32WStub
01003020 76f05790 GDI32!SetMapModeStub
01003024 76f060e0 GDI32!SetViewportExtExStub
01003028 76f060a0 GDI32!SetWindowExtExStub
0100302c 76f05540 GDI32!LPtoDPStub
01003030 76f03660 GDI32!DeleteObject
01003034 76f07950 GDI32!GetTextMetricsW
01003038 76f06370 GDI32!EndPage
0100303c 76f0b990 GDI32!AbortDoc
01003040 76f068c0 GDI32!EndDoc
01003044 76f03290 GDI32!DeleteDC
01003048 76f0b000 GDI32!SetAbortProc
0100304c 76f06840 GDI32!StartDocW
01003050 76f06340 GDI32!StartPage
01003054 76f06c60 GDI32!CreateDCW
01003058 76f04940 GDI32!CreateFontIndirectW
0100305c 76f047f0 GDI32!SetBkMode
01003060 76f041e0 GDI32!GetDeviceCaps
01003064 00000000
01003068 75450180 USER32!SetThreadDpiAwarenessContext
0100306c 7543e3f0 USER32!PostMessageW
01003070 75474170 USER32!DialogBoxParamW
01003074 7543ea10 USER32!GetFocus
01003078 75491370 USER32!MessageBoxW
0100307c 75432fe0 USER32!GetMenu
01003080 75446e40 USER32!CheckMenuItem
01003084 75447a10 USER32!GetSubMenu
01003088 75445f90 USER32!EnableMenuItem
0100308c 75455bd0 USER32!NtUserShowWindow
01003090 7543e8b0 USER32!GetDC
01003094 7543e290 USER32!ReleaseDC
01003098 754544dc USER32!SetCursorStub
0100309c 75451760 USER32!GetDpiForWindow
010030a0 75455840 USER32!NtUserSetActiveWindow
010030a4 7543f580 USER32!LoadStringW
010030a8 770882e0 ntdll!NtdllDefWindowProc_W
010030ac 7543b6f0 USER32!IsIconic
010030b0 75455980 USER32!NtUserSetFocus
010030b4 754543c0 USER32!PostQuitMessage
010030b8 75454ae0 USER32!NtUserDestroyWindow
010030bc 7549f1b0 USER32!MessageBeep
010030c0 75455470 USER32!NtUserMoveWindow
010030c4 75431cb0 USER32!GetDlgCtrlID
010030c8 75455b70 USER32!NtUserSetWindowPos
010030cc 754556c0 USER32!NtUserRedrawWindow
010030d0 7543f040 USER32!GetKeyboardLayout
010030d4 75453dc0 USER32!CharNextWStub
010030d8 7543fd40 USER32!SetWinEventHook
010030dc 7543e730 USER32!GetMessageW
010030e0 7544b790 USER32!TranslateAcceleratorW
010030e4 75437fe0 USER32!IsDialogMessageW
010030e8 754383c0 USER32!TranslateMessage
010030ec 75433eb0 USER32!DispatchMessageW
010030f0 75455c80 USER32!NtUserUnhookWinEvent
010030f4 75430620 USER32!SetWindowTextW
010030f8 7544fbd0 USER32!OpenClipboard
010030fc 7544f420 USER32!IsClipboardFormatAvailableStub
01003100 75450e80 USER32!CloseClipboardStub
01003104 75451380 USER32!SetDlgItemTextW
01003108 75450cf0 USER32!GetDlgItemTextW
0100310c 75450630 USER32!EndDialog
01003110 75440720 USER32!SendDlgItemMessageW
01003114 75441400 USER32!SetScrollPos
01003118 75455300 USER32!NtUserInvalidateRect
0100311c 75432f60 USER32!UpdateWindow
01003120 754551a0 USER32!NtUserGetWindowPlacement
01003124 75455b60 USER32!NtUserSetWindowPlacement
01003128 7543fca0 USER32!CharUpperWStub
0100312c 754550b0 USER32!NtUserGetSystemMenu
01003130 75453020 USER32!LoadAcceleratorsW
01003134 75438800 USER32!SetWindowLongW
01003138 7542f6c0 USER32!CreateWindowExW
0100313c 754306f0 USER32!MonitorFromWindow
01003140 7543f5b0 USER32!RegisterWindowMessageW
01003144 754313a0 USER32!LoadCursorW
01003148 7542e4c0 USER32!RegisterClassExW
0100314c 75430680 USER32!GetWindowTextLengthW
01003150 75433470 USER32!GetWindowLongW
01003154 75439da0 USER32!PeekMessageW
01003158 754312c0 USER32!GetWindowTextW
0100315c 7543fc80 USER32!EnableWindow
01003160 75451400 USER32!CreateDialogParamW
01003164 754531f0 USER32!DrawTextExW
01003168 75433b50 USER32!GetClientRect
0100316c 754342e0 USER32!SendMessageW
01003170 75454e10 USER32!NtUserGetForegroundWindow
01003174 7542e440 USER32!LoadIconW
01003178 75453220 USER32!LoadImageW
0100317c 00000000
01003180 7604ded0 advapi32!IsTextUnicode
01003184 00000000
01003188 76177de0 combase!CoCreateFreeThreadedMarshaler [onecore\com\combase\dcomrem\ipmrshl.cxx # 201]
0100318c 761a79d0 combase!CoWaitForMultipleHandles [onecore\com\combase\dcomrem\sync.cxx # 87]
01003190 761534c0 combase!PropVariantClear [onecore\com\combase\util\propvar.cxx # 278]
01003194 76182860 combase!CoTaskMemAlloc [onecore\com\combase\class\memapi.cxx # 428]
01003198 761440a0 combase!CoCreateGuid [onecore\com\combase\class\cocrguid.cxx # 49]
0100319c 761829e0 combase!CoTaskMemFree [onecore\com\combase\class\memapi.cxx # 444]
010031a0 76159910 combase!CoCreateInstance [onecore\com\combase\objact\actapi.cxx # 109]
010031a4 76195f60 combase!CoInitializeEx [onecore\com\combase\class\compobj.cxx # 3792]
010031a8 76195fb0 combase!CoUninitialize [onecore\com\combase\class\compobj.cxx # 3851]
010031ac 00000000
010031b0 74bcb460 KERNELBASE!GetTimeFormatW
010031b4 74bcf0d0 KERNELBASE!GetDateFormatW
010031b8 00000000
010031bc 74bc4bb0 KERNELBASE!IsDebuggerPresent
010031c0 74c0e2d0 KERNELBASE!wil::details::DebugBreak
010031c4 74c524b0 KERNELBASE!OutputDebugStringW
010031c8 00000000
010031cc 74c4d510 KERNELBASE!DelayLoadFailureHook
010031d0 00000000
010031d4 74bc1630 KERNELBASE!ResolveDelayLoadedAPI
010031d8 00000000
010031dc 74bc43a0 KERNELBASE!RaiseException
010031e0 74ba27a0 KERNELBASE!GetLastError
010031e4 77060240 ntdll!RtlSetLastWin32Error
010031e8 74bc0640 KERNELBASE!SetUnhandledExceptionFilter
010031ec 74c5b8f0 KERNELBASE!UnhandledExceptionFilter
010031f0 00000000
010031f4 74bbf770 KERNELBASE!FindClose
010031f8 74bc48d0 KERNELBASE!SetEndOfFile
010031fc 74b9ffb0 KERNELBASE!DeleteFileW
01003200 74bbc2d0 KERNELBASE!GetFullPathNameW
01003204 74ba1540 KERNELBASE!WriteFile
01003208 74b9f9e0 KERNELBASE!FindFirstFileW
0100320c 74bc0c50 KERNELBASE!GetFileAttributesExW
01003210 74b9f860 KERNELBASE!GetFileAttributesW
01003214 74ba20b0 KERNELBASE!CreateFileW
01003218 74ba1ee0 KERNELBASE!ReadFile
0100321c 74bc1750 KERNELBASE!GetFileInformationByHandle
01003220 00000000
01003224 74ba26f0 KERNELBASE!CloseHandle
01003228 00000000
0100322c 74b9b0b0 KERNELBASE!GetProcessHeap
01003230 7703ae50 ntdll!RtlAllocateHeap
01003234 7703dc70 ntdll!RtlFreeHeap
01003238 74bc4d20 KERNELBASE!HeapSetInformation
0100323c 00000000
01003240 74ba5210 KERNELBASE!LocalFree
01003244 74bbf030 KERNELBASE!GlobalAlloc
01003248 74ba5340 KERNELBASE!LocalAlloc
0100324c 74bb7c50 KERNELBASE!LocalUnlock
01003250 74b9b760 KERNELBASE!LocalReAlloc
01003254 74bb7aa0 KERNELBASE!LocalLock
01003258 74bbf820 KERNELBASE!GlobalFree
0100325c 00000000
01003260 757b4050 KERNEL32!GlobalLock
01003264 757b7b00 KERNEL32!LocalSize
01003268 757b4740 KERNEL32!GlobalUnlock
0100326c 00000000
01003270 74be0d80 KERNELBASE!MulDiv
01003274 00000000
01003278 74ba0420 KERNELBASE!GetModuleHandleW
0100327c 74ba1700 KERNELBASE!LoadLibraryExW
01003280 74b9ea60 KERNELBASE!GetProcAddress
01003284 74ba0320 KERNELBASE!FreeLibrary
01003288 74bbe8b0 KERNELBASE!GetModuleFileNameW
0100328c 74bbe7c0 KERNELBASE!GetModuleFileNameA
01003290 74b9f3b0 KERNELBASE!GetModuleHandleExW
01003294 00000000
01003298 74ba9a60 KERNELBASE!GetACP
0100329c 74b9b450 KERNELBASE!FormatMessageW
010032a0 74ba36e0 KERNELBASE!GetLocaleInfoW
010032a4 74c109a0 KERNELBASE!FindNLSString
010032a8 00000000
010032ac 74bbde70 KERNELBASE!GetUserDefaultUILanguage
010032b0 00000000
010032b4 74bbf740 KERNELBASE!UnmapViewOfFile
010032b8 74bbe6f0 KERNELBASE!MapViewOfFile
010032bc 74ba0910 KERNELBASE!CreateFileMappingW
010032c0 00000000
010032c4 74bc5000 KERNELBASE!GetCommandLineW
010032c8 00000000
010032cc 757b1a60 KERNEL32!GetCurrentThreadId
010032d0 757c3bf0 KERNEL32!GetCurrentProcess
010032d4 74bbf520 KERNELBASE!OpenProcessToken
010032d8 757c3c00 KERNEL32!GetCurrentProcessId
010032dc 757b9aa0 KERNEL32!GetStartupInfoWStub
010032e0 757bf420 KERNEL32!TerminateProcessStub
010032e4 00000000
010032e8 74bc0940 KERNELBASE!GetProcessMitigationPolicy
010032ec 00000000
010032f0 77060550 ntdll!RtlQueryPerformanceCounter
010032f4 00000000
010032f8 74bb4a20 KERNELBASE!RegOpenKeyExW
010032fc 74bb5800 KERNELBASE!RegCloseKey
01003300 74bb4460 KERNELBASE!RegQueryValueExW
01003304 74bd4530 KERNELBASE!RegSetValueExW
01003308 00000000
0100330c 7604efe0 advapi32!RegCreateKeyW
01003310 00000000
01003314 74b9f2c0 KERNELBASE!PathFileExistsW
01003318 74bd25a0 KERNELBASE!PathIsFileSpecW
0100331c 74bbebd0 KERNELBASE!PathFindExtensionW
01003320 00000000
01003324 74bb1de0 KERNELBASE!WideCharToMultiByte
01003328 74bd1490 KERNELBASE!FoldStringW
0100332c 74bb0390 KERNELBASE!CompareStringOrdinal
01003330 74ba6350 KERNELBASE!MultiByteToWideChar
01003334 00000000
01003338 757b7740 KERNEL32!lstrcmpiWStub
0100333c 00000000
01003340 770360b0 ntdll!RtlEnterCriticalSection
01003344 7705c380 ntdll!RtlDeleteCriticalSection
01003348 74ba0cb0 KERNELBASE!CreateMutexExW
0100334c 74bbeba0 KERNELBASE!SetEvent
01003350 77058680 ntdll!RtlLeaveCriticalSection
01003354 77053b40 ntdll!RtlReleaseSRWLockShared
01003358 74bbeac0 KERNELBASE!InitializeCriticalSectionEx
0100335c 74b9d850 KERNELBASE!OpenSemaphoreW
01003360 74bc1720 KERNELBASE!ReleaseSemaphore
01003364 74baef90 KERNELBASE!WaitForSingleObject
01003368 770384a0 ntdll!RtlAcquireSRWLockExclusive
0100336c 74baef10 KERNELBASE!ReleaseMutex
01003370 74b9f320 KERNELBASE!CreateSemaphoreExW
01003374 7703ec20 ntdll!RtlReleaseSRWLockExclusive
01003378 74baefb0 KERNELBASE!WaitForSingleObjectEx
0100337c 74bbe9c0 KERNELBASE!CreateEventExW
01003380 77053c10 ntdll!RtlAcquireSRWLockShared
01003384 00000000
01003388 74bbf590 KERNELBASE!Sleep
0100338c 77068e00 ntdll!RtlWakeAllConditionVariable
01003390 74bc3f70 KERNELBASE!SleepConditionVariableSRW
01003394 00000000
01003398 74bbe020 KERNELBASE!GetLocalTime
0100339c 74bbc6a0 KERNELBASE!GetSystemTimeAsFileTime
010033a0 74bb1a20 KERNELBASE!GetTickCount
010033a4 00000000
010033a8 77053ee0 ntdll!TpSetTimer
010033ac 77052580 ntdll!TpWaitForTimer
010033b0 770524f0 ntdll!TpReleaseTimer
010033b4 74bc3350 KERNELBASE!CreateThreadpoolTimer
010033b8 00000000
010033bc 761d39b0 combase!SetRestrictedErrorInfo [onecore\com\combase\winrt\error\restrictederror.cpp # 125]
010033c0 00000000
010033c4 761ddef0 combase!RoGetMatchingRestrictedErrorInfo [onecore\com\combase\winrt\error\restrictederror.cpp # 205]
010033c8 00000000
010033cc 7615ec90 combase!RoGetActivationFactory [onecore\com\combase\winrtbase\winrtbase.cpp # 1062]
010033d0 761d1b80 combase!RoInitialize [onecore\com\combase\winrtbase\winrtbase.cpp # 329]
010033d4 761db3e0 combase!RoUninitialize [onecore\com\combase\winrtbase\winrtbase.cpp # 454]
010033d8 00000000
010033dc 761626a0 combase!WindowsCreateStringReference [onecore\com\combase\winrt\string\string.cpp # 70]
010033e0 761bcf30 combase!WindowsDeleteString [onecore\com\combase\winrt\string\string.cpp # 146]
010033e4 7619c530 combase!WindowsGetStringRawBuffer [onecore\com\combase\winrt\string\string.cpp # 226]
010033e8 7614b690 combase!WindowsCreateString [onecore\com\combase\winrt\string\string.cpp # 30]
010033ec 00000000
010033f0 77044d10 ntdll!EtwEventUnregister
010033f4 77065d70 ntdll!EtwEventWriteTransfer
010033f8 7705e180 ntdll!EtwEventSetInformation
010033fc 7705f800 ntdll!EtwEventRegister
01003400 00000000
01003404 74ba10d0 KERNELBASE!GetTokenInformation
01003408 00000000
0100340c 756095b0 shcore!SHStrDupW
01003410 00000000
01003414 755ffea0 shcore!PathIsNetworkPathW
01003418 00000000
0100341c 75618a90 shcore!GetDpiForMonitor
01003420 00000000
01003424 74a46dd0 msvcrt!__dllonexit
01003428 74a373a0 msvcrt!free
0100342c 74a216c0 msvcrt!iswdigit
01003430 74aa5ba0 msvcrt!_acmdln
01003434 74a566f0 msvcrt!exit
01003438 74a88540 msvcrt!__setusermatherr
0100343c 74a56fe0 msvcrt!_unlock
01003440 74a25c50 msvcrt!__getmainargs
01003444 74a56e30 msvcrt!_lock
01003448 74a46eb0 msvcrt!_onexit
0100344c 74a561b0 msvcrt!_amsg_exit
01003450 74a7ab30 msvcrt!wcsnlen
01003454 74a3eb40 msvcrt!_ismbblead
01003458 74a47600 msvcrt!__set_app_type
0100345c 74a56230 msvcrt!_cexit
01003460 74a25d60 msvcrt!__p__commode
01003464 74a56110 msvcrt!_exit
01003468 74a44c40 msvcrt!_XcptFilter
0100346c 74a564c0 msvcrt!_initterm
01003470 74a77f60 msvcrt!_wcsicmp
01003474 74a23db0 msvcrt!_wtol
01003478 74a79910 msvcrt!memmove_s
0100347c 74a48bc0 msvcrt!_purecall
01003480 74a79500 msvcrt!memcpy_s
01003484 74a66ef0 msvcrt!_vsnwprintf
01003488 74a2b000 msvcrt!__CxxFrameHandler
0100348c 74a89fc0 msvcrt!_controlfp
01003490 74a2a670 msvcrt!terminate
01003494 74a48370 msvcrt!_except_handler4_common
01003498 74a25db0 msvcrt!__p__fmode
0100349c 74a37580 msvcrt!malloc
010034a0 74a34d70 msvcrt!_callnewh
010034a4 74a79130 msvcrt!memcmp
010034a8 74a79190 msvcrt!memcpy
010034ac 74a79970 msvcrt!memset
010034b0 00000000
010034b4 77088cb0 ntdll!LdrpValidateUserCallTarget
Reloading symbols made no difference:
0:007> .reload /f kernelbase.dll
SYMSRV: BYINDEX: 0x49
C:\symbols*https://msdl.microsoft.com/download/symbols
wkernelbase.pdb
017FA9C5278235B7E6BFBA74A9A5AAD91
SYMSRV: PATH: C:\symbols\wkernelbase.pdb\017FA9C5278235B7E6BFBA74A9A5AAD91\wkernelbase.pdb
SYMSRV: RESULT: 0x00000000
DBGHELP: KERNELBASE - public symbols
C:\symbols\wkernelbase.pdb\017FA9C5278235B7E6BFBA74A9A5AAD91\wkernelbase.pdb
I can set breakpoint via bm using wildcard, but in the past I have never had to do this:
0:000> bm kernelbase!RegOpenKeyExW*
1: 74bc64b0 #!"KERNELBASE!RegOpenKeyExW"
2: 74bb4a20 #!"KERNELBASE!RegOpenKeyExW"
Wondering if there is any specific change causing this, or is problem with symbols, etc?
well one is CLRTYPE private symbol I don't know how it crept in but iirc there are few more symbols like this
use .symopt+4000 to load only public symbols
then your breakpoint will be set correctly
0:000> .symopt
Symbol options are 0x30337:
0x00000001 - SYMOPT_CASE_INSENSITIVE
0x00000002 - SYMOPT_UNDNAME
0x00000004 - SYMOPT_DEFERRED_LOADS
0x00000010 - SYMOPT_LOAD_LINES
0x00000020 - SYMOPT_OMAP_FIND_NEAREST
0x00000100 - SYMOPT_NO_UNQUALIFIED_LOADS
0x00000200 - SYMOPT_FAIL_CRITICAL_ERRORS
0x00010000 - SYMOPT_AUTO_PUBLICS
0x00020000 - SYMOPT_NO_IMAGE_SEARCH
0:000> x /v /f /t kernelbase!RegOpenKeyExW*
prv func 00007fff`582a3120 6 <CLR type> KERNELBASE!RegOpenKeyExW (void)
pub func 00007fff`58248c60 0 <NoType> KERNELBASE!RegOpenKeyExW (<no parameter info>)
0:000> .symopt+4000
Symbol options are 0x34337:
0x00000001 - SYMOPT_CASE_INSENSITIVE
0x00000002 - SYMOPT_UNDNAME
0x00000004 - SYMOPT_DEFERRED_LOADS
0x00000010 - SYMOPT_LOAD_LINES
0x00000020 - SYMOPT_OMAP_FIND_NEAREST
0x00000100 - SYMOPT_NO_UNQUALIFIED_LOADS
0x00000200 - SYMOPT_FAIL_CRITICAL_ERRORS
0x00004000 - SYMOPT_PUBLICS_ONLY
0x00010000 - SYMOPT_AUTO_PUBLICS
0x00020000 - SYMOPT_NO_IMAGE_SEARCH
0:000> x /v /f /t kernelbase!RegOpenKeyExW*
pub func 00007fff`58248c60 0 <NoType> KERNELBASE!RegOpenKeyExW (<no parameter info>)
0:000> bp KERNELBASE!RegOpenKeyExW
0:000> bl
0 e Disable Clear 00007fff`58248c60 0001 (0001) 0:**** KERNELBASE!RegOpenKeyExW
0:000> g
ModLoad: 00007fff`59140000 00007fff`5916e000 C:\WINDOWS\System32\IMM32.DLL
Breakpoint 0 hit
KERNELBASE!RegOpenKeyExW:
00007fff`58248c60 4883ec38 sub rsp,38h
0:000> uf .
KERNELBASE!RegOpenKeyExW:
00007fff`58248c60 4883ec38 sub rsp,38h
00007fff`58248c64 488b442460 mov rax,qword ptr [rsp+60h]
00007fff`58248c69 488364242800 and qword ptr [rsp+28h],0
00007fff`58248c6f 4889442420 mov qword ptr [rsp+20h],rax
00007fff`58248c74 e817000000 call KERNELBASE!RegOpenKeyExInternalW (00007fff`58248c90)
00007fff`58248c79 4883c438 add rsp,38h
00007fff`58248c7d c3 ret
As stated there are quiet a few symbols that all point to this Address
windbg -c ".logopen d:\syms.txt;x /v /t kernelbase!*
;.logclose;q" windbg
D:\>wc -l syms.txt
41405 syms.txt
D:\>grep -i RegOpenKeyExW syms.txt
prv func 00007fff`582a3120 6 <CLR type> KERNELBASE!RegOpenKeyExW (void)
pub func 00007fff`58248c60 0 <NoType> KERNELBASE!RegOpenKeyExW (<no parameter info>)
D:\>grep -i 00007fff`582a3120 syms.txt | wc -l
1935
D:\>grep -i prv.*00007fff`582a3120 syms.txt | wc -l
1935
D:\>grep -i pub.*00007fff`582a3120 syms.txt | wc -l
0
I downloaded libhugetlbfs.so and have a simple test source :
int glbarr[1024*1024]={0} ;
int main()
{
char * ptr ;
ptr = (char *) malloc( 1024 * 1024 * 1 ) ;
printf(" press any key to go on \n");
getchar() ;
for(int idx=0;idx<100;idx++){
char strtmp[64] = {0} ;
sprintf(strtmp,"%020d",idx) ;
strcpy( ptr+1024*idx , strtmp ) ;
} //for
for(int idx=0;idx<100;idx++){
glbarr[idx] = idx ;
}
printf(" press any key to go on \n");
getchar() ;
} // main
then set the env :
export LD_PRELOAD=libhugetlbfs.so
export HUGETLB_MORECORE=yes
export HUGETLB_DEBUG=1
At last , execute test_malloc.exe :
INFO: Found pagesize 2048 kB
INFO: Detected page sizes:
INFO: Size: 2048 kB (default) Mount: /mnt/SharedMem_2M
INFO: Parsed kernel version: [3] . [10] . [0]
INFO: Feature private_reservations is present in this kernel
INFO: Feature noreserve_safe is present in this kernel
INFO: Feature map_hugetlb is present in this kernel
INFO: Kernel has MAP_PRIVATE reservations. Disabling heap prefaulting.
INFO: Kernel supports MAP_HUGETLB
INFO: HUGETLB_SHARE=0, sharing disabled
INFO: HUGETLB_NO_RESERVE=no, reservations enabled
INFO: Segment 0 (phdr 2): 0x400000-0x400a04 (filesz=0xa04) (prot = 0x5)
INFO: Segment 1 (phdr 3): 0x600de0-0xa01080 (filesz=0x274) (prot = 0x3)
DEBUG: symbol to copy at 0x601060: stdin
DEBUG: Total memsz = 0x400ca4, memsz of largest segment = 0x4002a0
INFO: libhugetlbfs version: 2.20
INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0xa04 bytes and 0 extra bytes from 0x400000...done
INFO: Prepare succeeded
INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0x274 bytes and 0x14 extra bytes from 0x600de0...done
INFO: Prepare succeeded
INFO: setup_morecore(): heapaddr = 0x1c00000
INFO: hugetlbfs_morecore(2101248) = ...
INFO: heapbase = 0x1c00000, heaptop = 0x1c00000, mapsize = 0, delta=2101248
INFO: Attempting to map 4194304 bytes
INFO: ... = 0x1c00000
INFO: hugetlbfs_morecore(0) = ...
INFO: heapbase = 0x1c00000, heaptop = 0x1e01000, mapsize = 400000, delta=-2093056
INFO: ... = 0x1e01000
and the /proc/pidof test_malloc.exe/maps :
00400000-00600000 r-xp 00000000 00:2b 6019488 /mnt/SharedMem_2M/libhugetlbfs.tmp.uI55WD (deleted)
00600000-00c00000 rw-p 00000000 00:2b 6123885 /mnt/SharedMem_2M/libhugetlbfs.tmp.VUALYM (deleted)
01c00000-02000000 rw-p 00000000 00:0d 6123886 /anon_hugepage (deleted)
numastat -m showes hugepage really in use for 8M ,
What bother me is what do "deleted" means in maps output ?!
either in /mnt/SharedMem_2M or /anon_hugepage
Edit :
and the debug information :
INFO: Found pagesize 2048 kB
INFO: Detected page sizes:
INFO: Size: 2048 kB (default) Mount: /mnt/SharedMem_2M
INFO: Parsed kernel version: [3] . [10] . [0]
INFO: Feature private_reservations is present in this kernel
INFO: Feature noreserve_safe is present in this kernel
INFO: Feature map_hugetlb is present in this kernel
INFO: Kernel has MAP_PRIVATE reservations. Disabling heap prefaulting.
INFO: Kernel supports MAP_HUGETLB
INFO: HUGETLB_SHARE=0, sharing disabled
INFO: HUGETLB_NO_RESERVE=no, reservations enabled
INFO: Segment 0 (phdr 3): 0x600de0-0xa01080 (filesz=0x274) (prot = 0x3)
DEBUG: symbol to copy at 0x601060: stdin
DEBUG: Total memsz = 0x4002a0, memsz of largest segment = 0x4002a0
INFO: libhugetlbfs version: 2.20
INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0x274 bytes and 0x14 extra bytes from 0x600de0...done
INFO: Prepare succeeded
INFO: setup_morecore(): heapaddr = 0x2200000
INFO: hugetlbfs_morecore(2101248) = ...
INFO: heapbase = 0x2200000, heaptop = 0x2200000, mapsize = 0, delta=2101248
INFO: Attempting to map 4194304 bytes
INFO: ... = 0x2200000
INFO: hugetlbfs_morecore(0) = ...
INFO: heapbase = 0x2200000, heaptop = 0x2401000, mapsize = 400000, delta=-2093056
INFO: ... = 0x2401000
and the maps :
00400000 default file=/home/marschen/test/posix-memalign/test_malloc.exe mapped=1 N0=1 kernelpagesize_kB=4
00600000 default file=/mnt/SharedMem_2M/libhugetlbfs.tmp.85Y41e\040(deleted) huge anon=1 dirty=1 N0=1 kernelpagesize_kB=2048
02200000 default file=/anon_hugepage\040(deleted) huge anon=1 dirty=1 N0=1 kernelpagesize_kB=2048
Temporary file deletion (unlinking) is normal for libhugetlbfs when it uses hugetlb fs pseudo filesystem (grep hugetlbfs /proc/filesystems) for getting mmaps backed with hugetlb pages.
For example, there is hugetlbfs_unlinked_fd function of libhugetlbfs/hugeutils.c
https://github.com/libhugetlbfs/libhugetlbfs/blob/e44180072b796c0e28e53c4d01ef6279caaa2a99/hugeutils.c#L1033
int hugetlbfs_unlinked_fd_for_size(long page_size)
{
const char *path;
char name[PATH_MAX+1];
int fd;
path = hugetlbfs_find_path_for_size(page_size);
..
name[sizeof(name)-1] = '\0';
strcpy(name, path);
strncat(name, "/libhugetlbfs.tmp.XXXXXX", sizeof(name)-1);
/* FIXME: deal with overflows */
fd = mkstemp64(name);
....
unlink(name);
return fd;
}
Temporary file name is randomly generated in mkstemp function; it also creates the file and opens it. Then this file is unlinked (man 2 unlink) from the filesystem (file name is marked as deleted in the directory, there is still inode and file data, but other programs can't access this file by name).
While unlinked fd is opened, it can be used to work with hugetlb mmap and to store data. Only when this fd is closed, file data will be actually deleted by fs.
Early unlinking of mktemp files is often used: When a file created with mkstemp() is deleted?
Some useful information can be also listed in HOWTO of libhugetlbfs project
https://github.com/libhugetlbfs/libhugetlbfs/blob/master/HOWTO
I wanna get some memory dump of a specific process.
I've found each windows process contain VadRoot in a EPROCESS.
I used windbg to get some information of this structure...
kd> dt nt!_MMVAD fffffa801b7011d0
+0x000 u1 : <unnamed-tag>
+0x008 LeftChild : (null)
+0x010 RightChild : (null)
+0x018 StartingVpn : 0x7fefe440
+0x020 EndingVpn : 0x7fefe4b0
+0x028 u : <unnamed-tag>
+0x030 PushLock : _EX_PUSH_LOCK
+0x038 u5 : <unnamed-tag>
+0x040 u2 : <unnamed-tag>
+0x048 Subsection : 0xfffffa80`19f62640 _SUBSECTION
+0x048 MappedSubsection : 0xfffffa80`19f62640 _MSUBSECTION
+0x050 FirstPrototypePte : 0xfffff8a0`00b3ac28 _MMPTE
+0x058 LastContiguousPte : 0xffffffff`fffffffc _MMPTE
+0x060 ViewLinks : _LIST_ENTRY [ 0xfffffa80`1b7a38c0 - 0xfffffa80`1aa6d6a0 ]
+0x070 VadsProcess : 0xfffffa80`1b7e8941 _EPROCESS
Its Win7 64bit.
I guess StartingVpn: 0x7fefe440 has contain the memory contents of this block.
But is that a virtual address? or physical address? i don't know
what it stands for...
Thanks.
locate process
lkd> !process 0 0 explorer.exe
PROCESS 8a1908d0 ...... Image: explorer.exe
set process context
lkd> .process /p /r 8a1908d0
view reqd module
lkd> lm m explorer
start end module name
01000000 010ff000 Explorer (deferred)
get the vadroot for a virtual address in the current process context
lkd> !vad explorer 1
VAD # 8a120ed0
Start VPN 1000 End VPN 10fe Control Area 8a81ab80
FirstProtoPte e23e9048 LastPte fffffffc Commit Charge 3 (3.)
Secured.Flink 0 Blink 0 Banked/Extend 0
File Offset 0
ImageMap ViewShare EXECUTE_WRITECOPY
ReadOnly
ControlArea # 8a81ab80
Segment e23e9008 Flink 00000000 Blink 00000000
Section Ref 1 Pfn Ref 4d Mapped Views 1
User Ref 2 WaitForDel 0 Flush Count 0
File Object 8ab28240 ModWriteCount 0 System Views 0
Flags (90000a0) Image File HadUserReference Accessed
\WINDOWS\explorer.exe
Segment # e23e9008
ControlArea 8a81ab80 BasedAddress 01000000
Total Ptes ff
WriteUserRef 0 SizeOfSegment ff000
Committed 0 PTE Template 8a81ac3000000420
Based Addr 1000000 Image Base 0
Image Commit 2 Image Info e23e9840
ProtoPtes e23e9048
Reload command: .reload explorer.exe=01000000,ff000
dump all the vads for a current process context
lkd> !vad 8a120ed0 0
VAD level start end commit
8a03b1d8 ( 3) e50 e51 0 Mapped READONLY Pagefile-backed section
8a6fe240 ( 4) e60 e6f 0 Mapped READWRITE Pagefile-backed section
................................
89c86600 ( 5) ff0 ff0 1 Private READWRITE
8a120ed0 ( 0) 1000 10fe 3 Mapped Exe EXECUTE_WRITECOPY \WINDOWS\explorer.exe
.................................
8a87bb18 ( 7) 26d0 2733 0 Mapped READWRITE \Documents and Settings\Admin\Local Settings\History\History.IE5\index.dat
8a74b420 ( 0) 3e1c0 3ec52 10 Mapped Exe EXECUTE_WRITECOPY \WINDOWS\system32\ieframe.dll
8abfa398 ( 1) 7ffde 7ffde 1 Private READWRITE
Total VADs: 1231, average level: 5, maximum depth: 4294967295
VAD is short for virtual address descriptor and VPN is short for virtual page number. So it's a virtual address, not a physical address.
It needs to be translated to a physical address using PTEs (page table entry).
Given a user mode address I found with a user mode debugging session:
0:032> !address
[...]
+ 7ff`fffdc000 7ff`fffde000 0`00002000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE TEB [~1; 13ec.10fc]
0:032> dd 7ff`fffde000 L8
000007ff`fffde000 00000000 00000000 00240000 00000000
000007ff`fffde010 0022b000 00000000 00000000 00000000
I can do this in a kernel debugging session using LiveKd (SysInternals):
0: kd> !process 0 0 explorer.exe
PROCESS fffffa8012ce5b10
SessionId: 1 Cid: 13ec Peb: 7fffffd6000 ParentCid: 13c0
DirBase: 3029e8000 ObjectTable: fffff8a006139d60 HandleCount: 1078.
Image: explorer.exe
0: kd> .process /p /r fffffa8012ce5b10
Implicit process is now fffffa80`12ce5b10
Loading User Symbols
[...]
0: kd> !vtop 0 7fffffde000
Amd64VtoP: Virt 000007ff`fffde000, pagedir 00000003`029e8000
Amd64VtoP: PML4E 00000003`029e8078
Amd64VtoP: PDPE 00000003`00ebcff8
Amd64VtoP: PDE 00000003`0203dff8
Amd64VtoP: PTE 00000003`01ebeef0
Amd64VtoP: Mapped phys 00000002`ff44f000
Virtual address 7fffffde000 translates to physical address 2ff44f000.
0: kd> dd /p 2ff44f000 L8
00000002`ff44f000 00000000 00000000 00240000 00000000
00000002`ff44f010 0022b000 00000000 00000000 00000000
Note how the content of the virtual address (dd) is identical to the physical address (dd /p).
Currently I am analyzing a dump with WinDbg.
I ran following commands (following Tess' incredible walkthrough):
~* e !clrstack
Which listed me all stacks of all threads. There are 300 running threads with more or less the same stack, so I am just printing the stack of one here ...
OS Thread Id: 0x107c (166)
Child SP IP Call Site
2bc1e654 77c1015d [HelperMethodFrame_1OBJ: 2bc1e654] System.Threading.WaitHandle.WaitOneNative(System.Runtime.InteropServices.SafeHandle, UInt32, Boolean, Boolean)
2bc1e720 6b2e6dd2 System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)
2bc1e73c 6b2e6d9c System.Threading.WaitHandle.WaitOne(Int32, Boolean)
2bc1e750 727f4baa System.Threading.ReaderWriterLockSlim.WaitOnEvent(System.Threading.EventWaitHandle, UInt32 ByRef, TimeoutTracker)
2bc1e78c 729bc154 System.Threading.ReaderWriterLockSlim.TryEnterUpgradeableReadLockCore(TimeoutTracker)
2bc1e7b8 725d250c System.Threading.ReaderWriterLockSlim.TryEnterUpgradeableReadLock(TimeoutTracker)
** MORE LINES **
Then I did the following:
> ~166s
eax=00000000 ebx=2bc1e444 ecx=00000000 edx=00000000 esi=00000001 edi=00000000
eip=77c1015d esp=2bc1e3f4 ebp=2bc1e490 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!NtWaitForMultipleObjects+0x15:
77c1015d 83c404 add esp,4
Then I was looking for all the instances in this thread:
> !dso
OS Thread Id: 0x107c (166)
ESP/REG Object Name
** MORE LINES **
2BC1E7D0 039ec48c System.Threading.ReaderWriterLockSlim
** MORE LINES **
Then I tried to get more information on the ReaderWriterLockSlim-instance:
> !do 039ec48c
Name: System.Threading.ReaderWriterLockSlim
MethodTable: 725ebda4
EEClass: 724543bc
Size: 68(0x44) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Core\v4.0_4.0.0.0__b77a5c561934e089\System.Core.dll
Fields:
MT Field Offset Type VT Attr Value Name
6b3b8138 4000755 3c System.Boolean 1 instance 0 fIsReentrant
6b3c3aa4 4000756 1c System.Int32 1 instance 0 myLock
6b3c7ae4 4000757 20 System.UInt32 1 instance 0 numWriteWaiters
6b3c7ae4 4000758 24 System.UInt32 1 instance 0 numReadWaiters
6b3c7ae4 4000759 28 System.UInt32 1 instance 0 numWriteUpgradeWaiters
6b3c7ae4 400075a 2c System.UInt32 1 instance 348 numUpgradeWaiters
6b3b8138 400075b 3d System.Boolean 1 instance 0 fNoWaiters
6b3c3aa4 400075c 30 System.Int32 1 instance 366 upgradeLockOwnerId
6b3c3aa4 400075d 34 System.Int32 1 instance 366 writeLockOwnerId
6b3c0ac0 400075e c ...g.EventWaitHandle 0 instance 00000000 writeEvent
6b3c0ac0 400075f 10 ...g.EventWaitHandle 0 instance 00000000 readEvent
6b3c0ac0 4000760 14 ...g.EventWaitHandle 0 instance 08188858 upgradeEvent
6b3c0ac0 4000761 18 ...g.EventWaitHandle 0 instance 00000000 waitUpgradeEvent
6b3b821c 4000763 4 System.Int64 1 instance 231 lockID
6b3b8138 4000765 3e System.Boolean 1 instance 0 fUpgradeThreadHoldingRead
6b3c7ae4 4000766 38 System.UInt32 1 instance 2147483649 owners
6b3b8138 4000767 3f System.Boolean 1 instance 0 fDisposed
6b3b821c 4000762 3e0 System.Int64 1 shared static s_nextLockID
>> Domain:Value 01742528:NotInit 01783fb8:NotInit 1268c9d8:NotInit <<
725fd46c 4000764 4 ...ReaderWriterCount 0 shared TLstatic t_rwc
>> Thread:Value <<
I've matched the value of owners (2147483649) against the the information on this page = 0x80000001
private const uint WRITER_HELD = 0x80000000;
private const uint WAITING_WRITERS = 0x40000000;
private const uint WAITING_UPGRADER = 0x20000000;
But how can I identify the thread which holds the lock?
If you have a deadlock, you can use SOSEx's !dlk command which does all the work for you.
*DEADLOCK DETECTED*
CLR thread 0x4 holds the Writer lock on ReaderWriterLockSlim 02712580
...and is waiting for a Reader lock on ReaderWriterLockSlim 0271253c
CLR thread 0x3 holds the Writer lock on ReaderWriterLockSlim 0271253c
...and is waiting for a Reader lock on ReaderWriterLockSlim 02712580
CLR Thread 0x4 is waiting at System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)(+0x1f IL,+0x22 Native)
CLR Thread 0x3 is waiting at System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)(+0x1f IL,+0x22 Native)
If you don't have a deadlock, you can still use SOSEx, but type !mlocks. The output looks like this, so you get all the different thread numbers and the type of the lock.
0:012> !mlocks
Examining SyncBlocks...
Scanning for ReaderWriterLock instances...
Scanning for holders of ReaderWriterLock locks...
Scanning for ReaderWriterLockSlim instances...
Scanning for holders of ReaderWriterLockSlim locks...
Examining CriticalSections...
ClrThread DbgThread OsThread LockType Lock LockLevel
----------------------------------------------------------------------
0x1 0 0x1460 thinlock 02718bcc (recursion:0)
0x5 6 0x1e80 RWLock 027125f0 Writer
0x8 9 0x22ac CritSect 027124c0
0x9 10 0x27b8 SyncBlock 0045f4e8
0xa 11 0x33f8 SyncBlock 0045f4b4
0x7 8 0x388c CritSect 027124a0
0x4 5 0x3d20 RWLockSlim 02712580 Writer
0x3 4 0x3e44 RWLockSlim 0271253c Writer
0x6 7 0x4704 RWLock 027125c4 Writer
If you have DML enabled, you can even click the links in the Lock column which gives more information. You can also type !rwlock <lock> if you don't like DML.
I did some more research, and decompiled ReaderWriterLockSlim and determined the meaning of writeLockOwnerId and upgradeLockOwnerId: They are actually the managed thread ids of the threads within the corresponding method.
Unfortunately you have to determine the ordinal of this managed thread id, by doing:
> !threads
ThreadCount: 371
UnstartedThread: 0
BackgroundThread: 371
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
Lock
ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Count Apt Exception
** MORE LINES **
381 366 1c0c 1e1fca78 1029220 Preemptive 3E46102C:00000000 01783fb8 2 MTA (Threadpool Worker)
** MORE LINES **
After this, you can simply do:
> ~381s
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=0000432d edi=4519d490
eip=77c0f8d1 esp=4519d448 ebp=4519d4b4 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!ZwWaitForSingleObject+0x15:
77c0f8d1 83c404 add esp,4
> !clrstack
OS Thread Id: 0x1c0c (381)
Child SP IP Call Site
** MORE LINES **
4519da08 69f35592 System.Data.SqlClient.SqlCommand.ExecuteReader(System.Data.CommandBehavior, System.String)
** MORE LINES **
Et voilà: the thread, which holds the upgradable lock, is currently executing some sql.
Consider the kernel module testhrarr.c given in debugging - Watch a variable (memory address) change in Linux kernel, and print stack trace when it changes? - Stack Overflow, which tracks write accesses to a memory location.
Now, I'm trying to track read/write accesses, which I've done by changing this line in the testhrarr_init function:
attr.bp_type = HW_BREAKPOINT_W | HW_BREAKPOINT_R;
First subquestion here is this: on my platform (Linux kernel 2.6.38); HW_BREAKPOINT_W on its own works, and HW_BREAKPOINT_W | HW_BREAKPOINT_R for read/write works too - however trying only read access with HW_BREAKPOINT_R doesn't work; as I get "Breakpoint registration failed" in /var/log/syslog, and insmod fails with "insmod: error inserting './testhrarr.ko': -1 Invalid parameters". Does anyone have any idea why?
Anyways, the problem here is that once I'm watching for read/write access with HW_BREAKPOINT_W | HW_BREAKPOINT_R, I cannot tell in my handler whether the stack trace I'm getting is due to a read or a write access. I have gone through:
struct perf_event_attr in include/linux/perf_event.h
struct hw_perf_event include/linux/perf_event.h
include/linux/hw_breakpoint.h for the HW_BREAKPOINT_* enum definitions
... and I couldn't find anywhere an explicit technique to retrieve from the hw breakpoint handler, whether it was a read or write access that triggered. I only found something in kernel/trace/trace_ksym.c, but trace_ksym has been deprecated (not even found in my 2.6.38), and besides it simply reads attr.bp_type, which is what we ourselves set in the kernel module.
So, after some brute-forcing, I realized that the hw_perf_event->interrupts may contain this information; so I modified the handler callback as:
static void sample_hbp_handler(struct perf_event *bp,
struct perf_sample_data *data,
struct pt_regs *regs)
{
struct perf_event_attr attr = bp->attr;
struct hw_perf_event hw = bp->hw;
char hwirep[8];
//it looks like printing %llu, data->type here causes segfault/oops when `cat` runs?
// apparently, hw.interrupts changes depending on read/write access (1 or 2)
// when only HW_BREAKPOINT_W, getting hw.interrupts == 1 always;
// only HW_BREAKPOINT_R - fails for me
// when both, hw.interrupts is either 1 or 2
// defined in include/linux/hw_breakpoint.h:
// HW_BREAKPOINT_R = 1, HW_BREAKPOINT_W = 2,
if (hw.interrupts == HW_BREAKPOINT_R) {
strcpy(hwirep, "_R");
} else if (hw.interrupts == HW_BREAKPOINT_W) {
strcpy(hwirep, "_W");
} else {
strcpy(hwirep, "__");
}
printk(KERN_INFO "+--- %s value is accessed (.bp_type %d, .type %d, state %d htype %d hwi %llu / %s ) ---+\n", ksym_name, attr.bp_type, attr.type, hw.state, hw.info.type, hw.interrupts, hwirep);
dump_stack();
printk(KERN_INFO "|___ Dump stack from sample_hbp_handler ___|\n");
}
... and this seems to work somewhat; as I get the following in syslog:
$ grep "testhrarr_arr_first value" /var/log/syslog
kernel: [ 200.887620] +--- testhrarr_arr_first value is accessed (.bp_type 3, .type 5, state 0 htype 131 hwi 1 / _R ) ---+
kernel: [ 200.892163] +--- testhrarr_arr_first value is accessed (.bp_type 3, .type 5, state 0 htype 131 hwi 1 / _R ) ---+
kernel: [ 200.892634] +--- testhrarr_arr_first value is accessed (.bp_type 3, .type 5, state 0 htype 131 hwi 2 / _W ) ---+
kernel: [ 200.912192] +--- testhrarr_arr_first value is accessed (.bp_type 3, .type 5, state 0 htype 131 hwi 1 / _R ) ---+
kernel: [ 200.912713] +--- testhrarr_arr_first value is accessed (.bp_type 3, .type 5, state 0 htype 131 hwi 2 / _W ) ---+
kernel: [ 200.932138] +--- testhrarr_arr_first value is accessed (.bp_type 3, .type 5, state 0 htype 131 hwi 1 / _R ) ---+
... however, if attr.bp_type is just HW_BREAKPOINT_W, then just three hw.interrupts == 1 are reported (which would be then wrongly reported as _R with the above code).
So then, if I just invert the meanings of _R and _W, I may get something that matches what I guess should occur - but this is quite obviously a shot in the dark, since I have no idea what hw_perf_event->interrupts should actually stand for.
So - does anyone know the proper way of determining the "direction" (read or write) of access to a hardware-watched memory location?
Edit: the answer for my first subquestion: for my architecture, x86, there is this piece of code:
http://lxr.free-electrons.com/source/arch/x86/kernel/hw_breakpoint.c?v=2.6.38#L252
static int arch_build_bp_info(struct perf_event *bp)
{
struct arch_hw_breakpoint *info = counter_arch_bp(bp);
info->address = bp->attr.bp_addr;
/* Type */
switch (bp->attr.bp_type) {
case HW_BREAKPOINT_W:
info->type = X86_BREAKPOINT_WRITE;
break;
case HW_BREAKPOINT_W | HW_BREAKPOINT_R:
info->type = X86_BREAKPOINT_RW;
break;
case HW_BREAKPOINT_X:
info->type = X86_BREAKPOINT_EXECUTE;
// ...
default:
return -EINVAL;
}
...
}
... which clearly states that X86 has either _WRITE or _RW breakpoints; so if we try to set up just for HW_BREAKPOINT_R, the process would fail returning -EINVAL.
So, I guess, I'd need the answer here primarily for X86, although if there is a generic portable mechanism for determining read/write access, I'd rather know about that...