All!
I'm debugging one quite strange case of process hanging/running out of memory using standard Windows crash dump with WinDbg. Obviously, it runs out of address space because of too many threads being created (it is 32 bit process), and I'm trying to figure out what's wrong with threads initialization (see callstack #3 below), because besides threads with callstacks that are typical for this program, it has handful of threads with callstacks of 3 types like:
1)
00 02cefb08 77544413 02fc024c 00000000 02cefb8c ntdll_774f0000!NtWaitForAlertByThreadId+0xc
01 02cefb28 7754434d 00000000 00000000 ffffffff ntdll_774f0000!RtlpWaitOnAddressWithTimeout+0x33
02 02cefb6c 7754423f 00000004 00000000 00000000 ntdll_774f0000!RtlpWaitOnAddress+0xa5
03 02cefba8 7752a605 02fc0000 02fc0000 02fc04b0 ntdll_774f0000!RtlpWaitOnCriticalSection+0xaa
04 02cefbc8 7752a525 02fc0248 02cefc88 77533844 ntdll_774f0000!RtlpEnterCriticalSectionContended+0xd5
05 02cefbd4 77533844 02fc0248 62da3da7 02fc04b0 ntdll_774f0000!RtlEnterCriticalSection+0x45
06 02cefc88 77533688 02fc04b0 02fc04b8 00000007 ntdll_774f0000!RtlpFreeHeap+0x174
07 02cefcd8 110d27fc 02fc0000 00000000 02fc04b8 ntdll_774f0000!RtlFreeHeap+0x758
...
These threads are stuck behind critical section 02fc024c that is taken by non-longer existing thread, and it is quite hard to figure out, what happened to it.
There are some threads that try to end normally, but are stuck in the LdrpDrainWorkQueue:
2)
# ChildEBP RetAddr Args to Child
00 05e5fd54 77527631 00000064 00000000 00000000 ntdll_774f0000!NtWaitForSingleObject+0xc
01 05e5fd78 7752b105 65f13f5f 00404e7c 00000000 ntdll_774f0000!LdrpDrainWorkQueue+0xbd
02 05e5fe70 7755179c 00404e7c 00404e7c 1086eb50 ntdll_774f0000!LdrShutdownThread+0x85
03 05e5ff40 00404efe 00000000 0042cef4 0042cefc ntdll_774f0000!RtlExitUserThread+0x4c
04 05e5ff6c 00404ea6 05e5ffcc 004049b8 05e5ff80 abc!EndThread+0x6
05 05e5ff80 743962c4 1086eb50 743962a0 941a355e abc!ThreadWrapper+0x2a
06 05e5ff94 77550779 1086eb50 65f13ef3 00000000 kernel32!BaseThreadInitThunk+0x24
07 05e5ffdc 77550744 ffffffff 77573606 00000000 ntdll_774f0000!__RtlUserThreadStart+0x2f
08 05e5ffec 00000000 00404e7c 1086eb50 00000000 ntdll_774f0000!_RtlUserThreadStart+0x1b
Also, dump presents about 1400 threads on a very early stage of initialization, that were created during last 5 minutes of process life with a callstack like:
3)
# ChildEBP RetAddr Args to Child
00 0ed1fba4 77527631 00000064 00000000 00000000 ntdll_774f0000!NtWaitForSingleObject+0xc
01 0ed1fbcc 7752b586 6ec53d9b ffffffff 1ed9d000 ntdll_774f0000!LdrpDrainWorkQueue+0xbd
02 0ed1fcb4 77557d86 6ec53c27 00000000 00000000 ntdll_774f0000!LdrpInitializeThread+0x8d
03 0ed1fd08 77557ce0 00000000 00000000 0ed1fd24 ntdll_774f0000!_LdrpInitialize+0x6a
04 0ed1fd10 00000000 0ed1fd24 774f0000 00000000 ntdll_774f0000!LdrInitializeThunk+0x10
These threads are also waiting in the LdrpDrainWorkQueue for event LdrpLoadCompleteEvent to be signalled.
This event is related to parallel loader (for some reference, first answer fot his question from RbMm, somewhat similar yet different situation here) This event is created during process initialization and signalled after parallel DLL loading has finished, so all LdrpInitializeThread's could traverse DllMain's and signal THREAD_ATTACH. But I don't understand why it is in non-signalled state on a process that has been running for weeks? Does parallel loader work on LoadLibrary as well, so LdrpLoadCompleteEvent gets reset? Couldn't find it in disassembly.
In any case, I'm trying to understand why process has developed such strange callstacks before it was forcefully terminated. I could imagine, that some thread began loading DLL that caused LdrpLoadCompleteEvent to be reset, then some thread holding lock for the heap died in a bad manner, so dll loading couldn't have been completed, so LdrpLoadCompleteEvent was never signalled, hence no new threads could have been initialized. However, there's no any thread that is loading dll in the dump.
Any insight/hint regarding how such callstacks could have been developed, or what else I could do to squeeze more info from the dump, is welcome.
Thank you!
Your fundamental problem is architectural ... a legendary problem known as "thrashing."
Your system is probably designed with what I refer to as the "flaming-arrow approach." Whenever a new request comes in, "just light another flaming arrow and throw it into the air." Unfortunately you just can't do that.
The permanent solution to your problem unfortunately will never be solved by a debugger: it will require a redesign.
Related
UPDATE:
So the question is as follows: My build setup generates an archive/lib (binary output), which I would like to extract some data from, for compression in my case, but that really is not the point.
My instincts tells me, that since a linker can extract constant data from an archive/lib, It should be possible/easy for me to "dump" the binary "contents" of a symbol contained in an archive/lib to e.g. a file....
So that's my question: How to dump binary contents of a symbol in an archive/lib (in ELF format)?
UPDATE:
I am building an application based on lvgl.
To allow texts I'm using the online tool provided by the lvgl maintainer to convert TrueType fonts into C-code (const data) which is linked into the application for rendering texts.
But the resulting data-set for the fonts is getting too large for my available flash memory, but I have a big chunk of un-used RAM. So, I would like to use heatshrink to compress the data, and un-compress to RAM at runtime.
This requires, that my build setup can extract the binary data, compress it, and link to flash, so that my run-time code can de-compress it.
I guessed, that I could stuff all the generated font-data into a "lib", extract the binary data, compress it, and link as a "blob" into the application.
But I'm failing to extract the data to compress from the library
E.g. my font-data declaration looks as follows:
/*Store the image of the letters (glyph)*/
static const uint8_t _glyph_bitmap[] =
{ /* const Byte values follow (e.g. 0x00) */ };
static const lv_font_glyph_dsc_t _glyph_dsc[] =
{ /* struct initialization follows */ }
lv_font_t myfont
{ /* struct initialization follows */ }
So I would need to access the myfont, and the declarations it is referencing in binary form.
I have a tool, which creates c-code representing some binary data, to allow the data to be compiled & linked into a final executable (ARM platform, GNU toolchain, custom hardware).
I am running out of flash, but have RAM to spare. So i'm considering compressing some large constant-data sections in a library, and decompress these to RAM as needed.
So I can compile the c-code, and stuff that into an archive. But so far i've had no luck trying to extract the binary data of the constant-data for compression using e.g. objdump or objcopy. But something tells me, that this is possible (and maybe easy even). But how? I've tried to "google" the problem, but came up empty-handed.
Eureka! I figured it out!
While I fully acknowledge and appreciate the advice given in comments / other answers, it still bothered me that I speculated it should be relatively easy to "play the linker" and extract blobs of hardcoded data from e.g an object-file.
Well, it turns out to be relatively easy (for elf format objects anyways) just as expected by using readelf
To dump a symbol, I used two steps:
Figure out the symbol index, by looking at the symbols in the object:
$ readelf --syms company_logo.o
Symbol table '.symtab' contains 17 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 00000000 0 NOTYPE LOCAL DEFAULT UND
1: 00000000 0 FILE LOCAL DEFAULT ABS company_logo.c
2: 00000000 0 SECTION LOCAL DEFAULT 1
3: 00000000 0 SECTION LOCAL DEFAULT 2
4: 00000000 0 SECTION LOCAL DEFAULT 3
5: 00000000 0 SECTION LOCAL DEFAULT 4
6: 00000000 0 NOTYPE LOCAL DEFAULT 4 $d
7: 00000000 0 SECTION LOCAL DEFAULT 6
8: 00000000 0 SECTION LOCAL DEFAULT 7
9: 00000000 0 SECTION LOCAL DEFAULT 9
10: 00000000 0 SECTION LOCAL DEFAULT 10
11: 00000000 0 SECTION LOCAL DEFAULT 12
12: 00000000 0 SECTION LOCAL DEFAULT 13
13: 00000000 0 SECTION LOCAL DEFAULT 14
14: 00000000 0 SECTION LOCAL DEFAULT 15
15: 00000000 12 OBJECT GLOBAL DEFAULT 4 company_logo
16: 00000000 21879 OBJECT GLOBAL DEFAULT 6 company_logo_map
Dump the contents of the symbol.
Now company_logo_mapwas my target, so use its index 6, as follows:
`readelf --hex-dump=6 company_logo.o`
` `
`Hex dump of section '.rodata.company_logo_map':`
` 0x00000000 00000000 00000000 00000000 00000000 ................`
` 0x00000010 00000000 00000000 00000000 00000000 ................`
` 0x00000020 00000000 00000000 00000000 00000000 ................`
` 0x00000030 00000000 00000000 00000000 00000000 ................`
` ... lots more data here`
Rather than extract data from the compiled binary, why not extract it from the generated C code, compress it, and generate equivalent C code with the compressed data?
This approach would probably simplify many aspects of implementation, debugging, and testing.
One of the users of my command line application has reported what appears to be an infinite loop. They helpfully took a dump of the process (via Task Manager) while it was in this state and sent it to me.
I'm not sure how to get useful information out of this dump. My normal technique of windbg -z the-dump-file.dmp -y releases\v5.0.0 -i releases\v5.0.0 doesn't give me much information that I know how to interpret. Are there ghc-specific tools I can use instead?
Moving forward, are the build options I should add or other things I should do to my release process to make this kind of post-mortem debugging more fruitful?
Here's an example of the stacks that I'm seeing. Not much useful info, especially for someone used to debugging C/C++ code in WinDbg. :-)
0 Id: 112dc.cc18 Suspend: 1 Teb: 00000000`00341000 Unfrozen
*** ERROR: Module load completed but symbols could not be loaded for gbc.exe
# Child-SP RetAddr Call Site
00 00000000`01b7d8d0 00000000`01049f71 gbc+0xc5676e
01 00000000`01b7d930 00000000`0104b5b4 gbc+0xc49f71
02 00000000`01b7d9a0 00000000`0104c644 gbc+0xc4b5b4
03 00000000`01b7da60 00000000`0104c1fa gbc+0xc4c644
04 00000000`01b7dab0 00000000`0042545b gbc+0xc4c1fa
05 00000000`01b7db30 00000000`011c40a0 gbc+0x2545b
06 00000000`01b7db38 00000000`0535bee1 gbc+0xdc40a0
07 00000000`01b7db40 00000000`010ffd80 0x535bee1
08 00000000`01b7db48 00000000`0535bee1 gbc+0xcffd80
09 00000000`01b7db50 00007ffb`3581fb01 0x535bee1
0a 00000000`01b7db58 00007ffb`3581b850 imm32!?MSCTF_NULL_THUNK_DATA_DLB+0x2e9
0b 00000000`01b7db60 00000000`00000010 imm32!CtfImmGetCompatibleKeyboardLayout
0c 00000000`01b7db68 00000000`00000000 0x10
1 Id: 112dc.d324 Suspend: 1 Teb: 00000000`00349000 Unfrozen
# Child-SP RetAddr Call Site
00 00000000`05c2fc48 00007ffb`36441563 ntdll!ZwWaitForWorkViaWorkerFactory+0x14
01 00000000`05c2fc50 00007ffb`34172774 ntdll!TppWorkerThread+0x293
02 00000000`05c2ff60 00007ffb`36470d61 kernel32!BaseThreadInitThunk+0x14
03 00000000`05c2ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21
2 Id: 112dc.11b48 Suspend: 1 Teb: 00000000`0034b000 Unfrozen
# Child-SP RetAddr Call Site
00 00000000`0642dd38 00007ffb`32f2988f ntdll!ZwWaitForSingleObject+0x14
01 00000000`0642dd40 00000000`00ffca15 KERNELBASE!WaitForSingleObjectEx+0x9f
02 00000000`0642dde0 00000000`00000000 gbc+0xbfca15
Some resources that might be useful. (If there are more up-to-date ones, I would like to see them myself.)
https://ghc.haskell.org/trac/ghc/wiki/Debugging/CompiledCode
https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/debug-info.html
https://wiki.haskell.org/Debugging
A few important nuggets:
The runtime flag +RTS -? Will tell you what runtime flags add debugging information. These will start with +RTS -D. For example, +RTS -DS turns on a number of runtime assertions and sanity checks.
The strange names you see are encoded in something called Z-encoding. This is defined at https://ghc.haskell.org/trac/ghc/browser/ghc/compiler/cmm/CLabel.hs.
If you can recompile the code with debugging symbols on and threading off, and still reproduce the bug, you can set breakpoints (or hit control-C) inside the debugger and backtrace from there. You can examine memory with a command like print/a 0x006eb0c0 (although you seem to be using 64-bit pointers). You can see the assembly-language instruction that crashed with disassemble.
You need to use the -ddump-stg compile flag to see what the variable names mean, because that is the last phase of the transformation before the program is assembled, and the variable names you see in the debugger correspond to the ones here.
You can instrument the code with Debug.Trace.
I'm trying to analyze a BSOD (0xFC,ATTEMPTED_EXECUTE_OF_NOEXECUTE_MEMORY) which occurred on a Windows 10 machine during a restart, but I'm stuck.
!analyze -v gave useful information like:
DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT
PROCESS_NAME: System
MODULE_NAME: mrxsmb
IMAGE_NAME: mrxsmb.sys
FAILURE_BUCKET_ID: 0xFC_mrxsmb!MRxSmbQueryLbfoTeamCapability
BUCKET_ID: 0xFC_mrxsmb!MRxSmbQueryLbfoTeamCapability
PRIMARY_PROBLEM_CLASS: 0xFC_mrxsmb!MRxSmbQueryLbfoTeamCapability
# Child-SP RetAddr Call Site
00 ffffd000`20505ba8 fffff802`a03e1a7c nt!KeBugCheckEx
01 ffffd000`20505bb0 fffff802`a0319229 nt! ?? ::FNODOBFM::`string'+0x1377c
02 ffffd000`20505bf0 fffff802`a031a661 nt!MiSystemFault+0x7e9
03 ffffd000`20505ce0 fffff802`a03c8bbc nt!MmAccessFault+0x5f1
04 ffffd000`20505e40 fffff800`1cf70030 nt!KiPageFault+0x13c
05 ffffd000`20505fd8 fffff802`a072fea2 ndis!_NULL_IMPORT_DESCRIPTOR+0x558
06 ffffd000`20505fe0 fffff802`a07300e6 nt!IopXxxControlFile+0x1162
07 ffffd000`205061b0 fffff802`a03ca1a3 nt!NtDeviceIoControlFile+0x56
08 ffffd000`20506220 fffff802`a03c2860 nt!KiSystemServiceCopyEnd+0x13
09 ffffd000`20506428 fffff800`1d7df2aa nt!KiServiceLinkage
0a ffffd000`20506430 fffff800`1d818b5b mrxsmb!MRxSmbQueryLbfoTeamCapability+0x16a
0b ffffd000`20506570 fffff800`1cbc1c7d mrxsmb!MRxSmbPnPBindingHandler+0x1ab
0c ffffd000`205066e0 fffff800`1cbc2194 TDI!TdiNotifyNewPnpClient+0x12d
0d ffffd000`20506710 fffff800`1cbc1db0 TDI!TdiExecuteRequest+0x2d4
0e ffffd000`20506760 fffff800`1cbc184c TDI!TdiHandleSerializedRequest+0x120
0f ffffd000`205067d0 fffff800`1d7e0b96 TDI!TdiRegisterPnPHandlers+0x12c
10 ffffd000`20506800 fffff800`1d81ab71 mrxsmb!MRxSmbStartTransport+0xfe
11 ffffd000`20506890 fffff800`1d228f2f mrxsmb!MRxSmbStart+0x51
12 ffffd000`205068e0 fffff800`1d816faf rdbss!RxStartMinirdr+0xef
13 ffffd000`20506930 fffff800`1d7d366e mrxsmb!MRxSmbDevFcbXXXControlFile+0x1eb
14 ffffd000`205069a0 fffff800`1d2567de mrxsmb!SmbShellDevFcbXXXControlFile+0xe
15 ffffd000`205069d0 fffff800`1d256665 rdbss!RxXXXControlFileCallthru+0xb6
16 ffffd000`20506a00 fffff800`1d2694d9 rdbss!RxCommonDevFCBFsCtl+0xa5
17 ffffd000`20506a50 fffff802`a0367c11 rdbss!RxFspDispatch+0xe9
18 ffffd000`20506b10 fffff802`a034f399 nt!IopProcessWorkItem+0x81
19 ffffd000`20506b80 fffff802`a02c3b35 nt!ExpWorkerThread+0xe9
1a ffffd000`20506c10 fffff802`a03c4b16 nt!PspSystemThreadStartup+0x41
1b ffffd000`20506c60 00000000`00000000 nt!KiStartSystemThread+0x16
These info gave me some idea what went wrong:
The source of the problem might be in the mrxsmb!MRxSmbQueryLbfoTeamCapability function according to WinDbg.
MRxSMB is the so called "SMB Mini-Redirector". A redirector is a component which is part of a remote file system driver. The rdbss stuff is also part of this redirector architecture which MRxSMB is built on.
The TDI stuff stands for "Transport Driver Interface" and is according to the Windows Internals book some sort of legacy "kernel-mode device driver that implements the kernel-mode portion of a networking API's implementation".
From that I assume Windows tries to access a remote filesystem via SMB. I think it actually tries to access a network drive which is not accessible.
But I'd like to understand why this causes a BSOD.
My questions are:
How does WinDbg know that the problem starts with mrxsmb!MRxSmbQueryLbfoTeamCapability+0x16a and doesn't occur later?
ndis!_NULL_IMPORT_DESCRIPTOR looks suspicious. According to google such a descriptor marks the end of a loaded library. But why does it then show up as function here?
If I take the return address of ndis!_NULL_IMPORT_DESCRIPTOR (fffff802`a072fea2) and go some bytes back in memory I get this:
1: kd> u fffff802`a072fe90 fffff802`a072fea2
nt!IopXxxControlFile+0x1150:
fffff802`a072fe90 eb13 jmp nt!IopXxxControlFile+0x1165 (fffff802`a072fea5)
fffff802`a072fe92 488bd7 mov rdx,rdi
fffff802`a072fe95 488b8c2488000000 mov rcx,qword ptr [rsp+88h]
fffff802`a072fe9d e8fe8dc1ff call nt!IofCallDriver (fffff802`a0348ca0)
fffff802`a072fea2 448be0 mov r12d,eax
Is my assumption correct, that the ndis!_NULL_IMPORT_DESCRIPTOR+0x558 thing happens actually in nt!IofCallDriver since that is the last instruction before the return address of ndis!_NULL_IMPORT_DESCRIPTOR+0x558? But why does this function then not show up in the stack trace?
How can I actually get function parameters? From my understanding I can't because the calling convention is that the first 4 parameters are passed in registers rather than the stack and I only have the register contents of the last operation?
How to proceed?
The question may be a bit awkward, but here's my detailed problem:
Currently I'm looking into setting up SysInternals' procdump.exe to monitor an application of ours that exhibits spurious disappearances -- that is, the user reports that the application is simply "gone" without any trace after a short visible hang of the application's window.
My first idea was to run procdump -e -x . MyApp.exe which would record a crash dump when the application encounters an unhandled exception, but then I saw that there is also a -t switch, that --
-t - Write a dump when the process terminates.
automatically generates a dump when the process terminates.
Now the problem
I have tested the -t switch with our app by inserting a ExitProcess or TerminateProcess call at a defined location where I can trigger it.
While the app behaves as expected, i.e. TerminateProcess immediately "kills" the running app and ExitProcess takes a while because global cleanup is run, the dump generated this way is useless in both cases.
The dumps I get for -t always contain only a sinlge thread (where the app was running over 20 thread at termination time) and the callstack isn't even at a useful location. (It just seems to be one random thread from the terminated app.)
Am I doing something wrong? Can I usefully use procdump -t to track down unexpected calls of process exit functions at all?
Can I usefully use procdump -t to track down unexpected calls of
process exit functions at all?
I think not and here's why:
test process calc.exe
CommandLine: "C:\Program Files\Sysinternals\procdump.exe" -t calc.exe
I try to carefully suggest that procdump is waiting on calc.exe process handle.
0:000> kb
ChildEBP RetAddr Args to Child
0017f2e0 77135e6c 75336872 00000002 0017f334 ntdll!KiFastSystemCallRet
0017f2e4 75336872 00000002 0017f334 00000001 ntdll!NtWaitForMultipleObjects+0xc
0017f380 76cbf14a 0017f334 0017f3a8 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100
0017f3c8 76cbf2c2 00000002 7ffdb000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0017f3e4 011c6135 00000002 0017f46c 00000000 kernel32!WaitForMultipleObjects+0x18
WARNING: Stack unwind information not available. Following frames may be wrong.
0017fc30 011c999e 00000003 013d1de0 013d1e78 procdump+0x6135
0017fc78 76cc1194 7ffdb000 0017fcc4 7714b495 procdump+0x999e
0017fc84 7714b495 7ffdb000 77ad79b5 00000000 kernel32!BaseThreadInitThunk+0xe
0017fcc4 7714b468 011c99f5 7ffdb000 00000000 ntdll!__RtlUserThreadStart+0x70
0017fcdc 00000000 011c99f5 7ffdb000 00000000 ntdll!_RtlUserThreadStart+0x1b
0:000> dd 17f46c
0017f46c 00000238 00000268
0:000> !handle 238 f
Handle 238
Type Process
Attributes 0
GrantedAccess 0x1fffff:
Delete,ReadControl,WriteDac,WriteOwner,Synch
Terminate,CreateThread,,VMOp,VMRead,VMWrite,DupHandle,CreateProcess,SetQuota,SetInfo,QueryInfo,SetPort
HandleCount 5
PointerCount 52
Name <none>
Object Specific Information
Process Id 1580
Parent Process 2476
Base Priority 8
In the crash dump file gets stack last complete process thread (TID 3136) just before the end of the process.
0:000> ~
. 0 Id: dc8.c40 Suspend: -1 Teb: 7ffdd000 Unfrozen
0:000> .formats c40
Evaluate expression:
Hex: 00000c40
Decimal: 3136
Crash dump file is created after the completion of the last thread, and before the end of the process.
I am debugging a managed application using Son of Strike (SOS) in Visual studio 2010. I want to run a raw memory dump from a specific location, but I get "End of expression expected" error. If I attach WinDbg, then I can run same 'dd' command. How can I fix this problem?
!clrstack -l
OS Thread Id: 0xd5c (3420)
Child SP IP Call Site
0050eeac 002700eb ConsoleApplication2.Program.Main(System.String[])
LOCALS:
0x0050eeb0 = 0x0240c178
0x0050eebc = 0x00000000
0050f0fc 6b4c21bb [GCFrame: 0050f0fc]
dd 0x0240c178
End of expression expected
dd 0x0050eeb0
End of expression expected
In the Immediate window you have to use >dd 0x001AF2E0 to make it work. You have to type the > before dd.
dd 0x001AF2E0
End of expression expected
>dd 0x001AF2E0
0x001AF2E0 6d7c4938 ffffffff 001af34c 00000001
0x001AF2F0 002dd780 00000000 002dd780 ffffffff
0x001AF300 00000001 77a220f9 00000000 00713000
0x001AF310 002711a8 00000001 00000000 00000000
In the Command window you can just type dd 0x001AF2E0.
Type .cordll and see if the sos dll is loaded.
eg:
0:000> .cordll
CLR DLL status: Loaded DLL C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll