The following message is copied from WinDBG's output window:
Use !analyze -v to get detailed debugging information.
BugCheck 24, {1904fb, 8f9ec9f8, 8f9ec5d0, 87c4fccc}
Probably caused by : Ntfs.sys ( Ntfs!NtfsCommonRead+6f4 )
Followup: MachineOwner
Please note the last line: "Followup: MachineOwner"
What does it mean?
The Followup output is the attempt by the !analyze command to triage the exception to identify who owns the fault, sometimes it could be a driver, hardware, application etc..
There is a default entry in the triage.ini file that !analyze (and also !owner command) uses when it cannot find a match:
default=MachineOwner
This file is located in a folder called triage on my machine it is in
C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\triage
It is something you can ignore in your case, there is more information about how the analysis works and how to edit the triage.ini file for your specific use case.
Related
I'm trying to debug an issue with what may be handle leak. I have a dump created on a remote windows machine and I would like to see the handles information. I'm using WinDbg. I have seen some articles from the MSDN and from other sources, like https://www.codeproject.com/Articles/6988/Debug-Tutorial-Part-5-Handle-Leaks, but I can't get it to work, so I need some help.
I tried the next
Using !handles or !devhandles - these have failed. I either get no export handles found or a failure to load kdexts. Apparently kernel debugging is not enabled.
I found winxp/kdexts.dll in my path (given from .chain command) but it wouldn't load - .load kdexts yields `DebugExtensionInitializeFailed1 with error code 0x80004005.
Using !handle - with !handle -? I get help for the command but when I try something else, I get "Unable to read handle information". For example,
!handle - I expected a full list of handles
!handle 0 0
!handle 0 0 file
My setup
The remote process is Windows server 2012 (64bit), just as my own machine
I'm using the latest WinDbg from the windows sdk 10
I have a full dump, created by right-clicking task manager
I need some help if possible
Do I need to do kernel-debugging in order to view the list of handles from the dump?
Is it possible at all to do kernel-debugging on a full dump created from task-manager? Or is it required that the dump be taken differently?
How can I know if a given dump file includes the handle information?
How can I use the !handle command properly?
Is there any other alternative, such as using visual studio, another utility, etc.?
I'd appreciate any help
!tamir
your dump was probably a dump taken without handle information
you may use dumpchk.exe that comes with windbg installation to see if Handle Stream exists in the dump
if you have control over dump creation check how to use .dump /ma with windbg
or you may also explore sysinternals procdump.exe
and also make sure you are using the correct bitted debugger for the dump in question
a sample path
D:\>dir /s /b "c:\Program Files (x86)\Windows Kits\10\Debuggers\cdb.exe"
c:\Program Files (x86)\Windows Kits\10\Debuggers\arm\cdb.exe
c:\Program Files (x86)\Windows Kits\10\Debuggers\arm64\cdb.exe
c:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe
c:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe
here is a sample dump creation with and without handle stream in the dump
:000> .dump /ma d:\madump.dmp
Creating d:\madump.dmp - mini user dump
Dump successfully written
0:000> .dump d:\nomadump.dmp
Creating d:\nomadump.dmp - mini user dump
Dump successfully written
0:000> q
analysing both the dumps with dumpchk and checking for streams present
dumpchk nomadump.dmp > nomachk.txt
dumpchk madump.dmp > machk.txt
D:\>type machk.txt |grep -i number.*stream
NumberOfStreams 17
D:\>type nomachk.txt |grep -i number.*stream
NumberOfStreams 13
diff
D:\>diff -y machk.txt nomachk.txt
Microsoft (R) Windows Debugger Version 10.0.17763.132 AMD64 Microsoft (R) Windows Debugger Version 10.0.17763.132 AMD64
Loading Dump File [D:\madump.dmp] | Loading Dump File [D:\nomadump.dmp]
User Mini Dump File with Full Memory: Only application d | User Mini Dump File: Only registers, stack and portions of me
----- User Mini Dump Analysis ----- User Mini Dump Analysis
MINIDUMP_HEADER: MINIDUMP_HEADER:
Version A793 (A063) Version A793 (A063)
NumberOfStreams 17 | NumberOfStreams 13
Flags 441826 | Flags 40000
0002 MiniDumpWithFullMemory <
0004 MiniDumpWithHandleData <
0020 MiniDumpWithUnloadedModules <
0800 MiniDumpWithFullMemoryInfo <
1000 MiniDumpWithThreadInfo <
40000 MiniDumpWithTokenInformation 40000 MiniDumpWithTokenInformation
400000 MiniDumpWithIptTrace <
if you feel enterprising take a look here for some hints to deciphering a dump without windbg /dbgeng
forgot to post the result of doing !handle on both dumps
D:\>cdb -c "!handle;q" -z nomadump.dmp |awk /Reading/,/quit/"
0:000> cdb: Reading initial command '!handle;q'
ERROR: !handle: extension exception 0x80004002.
"Unable to read handle information"
quit:
D:\>cdb -c "!handle;q" -z madump.dmp |awk /Reading/,/quit/"
0:000> cdb: Reading initial command '!handle;q'
Handle 0000000000000004
Type File
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxSNIPxxxxxxxxx
Handle 0000000000000128
Type Mutant
Handle 000000000000012c
Type
Handle 0000000000000180
Type File
70 Handles
Type Count
None 27
Event 13
File 8
Directory 2
Mutant 1
Semaphore 2
Key 6
IoCompletion 2
TpWorkerFactory 2
ALPC Port 1
WaitCompletionPacket 6
quit:
Check the tool which was used for creating the crash dump. Perhaps it provides an option to include handle data.
Task Manager includes handle data by default
Visual Studio includes handle data by default
In WinDbg, .dump can be used with the /mh switch to include handle data. /ma is a shortcut for /mfFhut, so it also includes handle data.
ProcDump automatically includes handle data.
Windows Error Reporting LocalDumps can be configured with a Registry value called CustomDumpFlags.
If you create the dump programmatically yourself with MiniDumpWriteDump(), use MINIDUMP_TYPE::MiniDumpWithHandleData.
I have been trying to get a w3wp crash dump to see the crash callstack. I got two dumps but both of them have a single thread in them - seems almost like AppDomain has been recycled already and there is nothing useful left in the process when the dump was saved.
Command used: "procdump -mm -e -n 1 -l pt <PID>"
Also tried -ma for full dump but result is the same:
0:000> ~
. 0 Id: fa4.1dc8 Suspend: -1 Teb: 000000b1`77a78000 Unfrozen
I am not sure if I am missing something in the command, or IIS does not provide usable managed dumps when capturing them with procdump - any inputs are highly appreciated!
Additional detail: I was seeing STACK_OVERFLOW exception being logged by the procdump, which - apparently needs different method to capture useful dump. See my own answer below for details.
It only took few hours - hopefully this will save some time for others like me.
Found a way to do this:
procdump -mm -e 1 -l -f C00000FD.STACK_OVERFLOW -g <PID>
It works! Thanks to the unknown fellow member whose hint lead me to this. I was reading too many pages and missed saving the page link to post acknowledgement here.
I'm trying to understand the mechanics of loading an executable file, so I did two different tests with notepad.exe
1) Running dumpbin command:
dumpbin /ALL "C:\Windows\System32\notepad.exe" /OUT:"C:\sample\log4.txt"
I got the following values under OPTIONALHEADER VALUES:
1AC50 entry point (000000014001AC50) WinMainCRTStartup
1000 base of code
140000000 image base (0000000140000000 to 0000000140042FFF)
2) Running WinDbg:
x notepad!*CRT*
I got these:
00b9bf9a notepad!__mainCRTStartup (void)
00b9bf90 notepad!WinMainCRTStartup (<no parameter info>)
00ba04a4 notepad!msvcrt_NULL_THUNK_DATA = <no type information>
00ba050c notepad!_IMPORT_DESCRIPTOR_msvcrt = <no type information>
I don't understand why 14001AC50 and 00b9bf90 are different values. Shouldn't they be the same AddressOfEntryPoint value?
Thanks in advance
There are a couple reasons for the discrepancy.
First, you are running dumpbin on the x64 version of notepad.exe, stored in System32 but you seem to be debugging the x86 notepad.exe stored in SysWoW64. Make sure you've launched the x64 or AMD64 version of WinDbg and that you're attaching to C:\Windows\System32\notepad.exe.
Once that's sorted out things should start making more sense but there's one more thing to keep in mind. The x command in WinDbg is displaying the virtual memory address of the symbol in the running process while dumpbin displays it as an offset from the module base address.
Some quick subtraction from the module base and things should match up.
Here's how it looks on my system:
C:\>dumpbin /ALL "C:\Windows\System32\notepad.exe" | find "entry point"
1AC50 entry point (000000014001AC50) WinMainCRTStartup
0:000> x notepad!WinMainCRTStartup
00007ff6`4fe1ac50 notepad!WinMainCRTStartup (<no parameter info>)
0:000> ? notepad!WinMainCRTStartup - notepad
Evaluate expression: 109648 = 00000000`0001ac50
After running a set of tests with drmemory overnight I am trying to resolve the error stacks by providing pdb symbols. The pdb's come from a large samba-mapped repository and using _NT_SYMBOL_PATH at runtime slowed things down too much.
Does anyone know of a tool that post-processes results.txt and pulls new symbols (via NT_SYMBOL_PATH or otherwise) as required to produce more detailed stacks ? If not, any hints for adapting asan_symbolize.py to do this ?
https://llvm.org/svn/llvm-project/compiler-rt/trunk/lib/asan/scripts/asan_symbolize.py
What I came up with so far using dbghelp.dll is below. Works but could be better.
https://github.com/patraulea/postpdb
ok this Query does not pertain to use of windbg or doesn't have anything to do with _NT_SYMBOL_PATH
Dr.Memory is a memory diagnostic tool akin to valgrind and is based on Dynamorio instumentation framework usable on raw unmodified binaries
on windows you can invoke it like drmemory.exe calc.exe from a command prompt (cmd.exe)
as soon as the binary finishes execution a log file named results.txt is written to a default location
if you had setup _NT_SYMBOL_PATH drmemory honors it and resolves symbol information from prepulled symbol file (viz *.pdb) it does not seem to download files from ms symbol server it simply seems to ignore the SRV* cache and seems to use only the downstream symbol folder
so if the pdb file is missing or isnt downloaded yet
the results.txt will contain stack trace like
# 6 USER32.dll!gapfnScSendMessage +0x1ce (0x75fdc4e7 <USER32.dll+0x1c4e7>)
# 7 USER32.dll!gapfnScSendMessage +0x2ce (0x75fdc5e7 <USER32.dll+0x1c5e7>)
while if the symbol file was available it would show
# 6 USER32.dll!InternalCallWinProc
# 7 USER32.dll!UserCallWinProcCheckWow
so basically you need the symbol file for appplication in question
so as i commented you need to fetch the symbols for the exe in question
you can use symchk on a running process too and create a manifest file
and you can use symchk on a machine that is connected to internet
to download symbols and copy it to a local folder on a non_internet machine
and point _NT_SYMBOL_PATH to this folder
>tlist | grep calc.exe
1772 calc.exe Calculator
>symchk /om calcsyms.txt /ip 1772
SYMCHK: GdiPlus.dll FAILED - MicrosoftWindowsGdiPlus-
1.1.7601.17514-gdiplus.pdb mismatched or not found
SYMCHK: FAILED files = 1
SYMCHK: PASSED + IGNORED files = 27
>head -n 4 calcsyms.txt
calc.pdb,971D2945E998438C847643A9DB39C88E2,1
calc.exe,4ce7979dc0000,1
ntdll.pdb,120028FA453F4CD5A6A404EC37396A582,1
ntdll.dll,4ce7b96e13c000,1
>tail -n 4 calcsyms.txt
CLBCatQ.pdb,00A720C79BAC402295B6EBDC147257182,1
clbcatq.dll,4a5bd9b183000,1
oleacc.pdb,67620D076A2E43C5A18ECD5AF77AADBE2,1
oleacc.dll,4a5bdac83c000,1
so assuming you have fetched the symbols it would be easier to rerun the tests with a locally cached copies of the symbol files
if you have fetched the symbols but you cannot rerun the tests and have to work solely with the output from results.txt you have some text processing work (sed . grep , awk . or custom parser)
the drmemory suite comes with a symbolquery.exe in the bin folder and it can be used to resolve the symbols from results.txt
in the example above you can notice the offset relative to modulebase like
0x1c4e7 in the line # 6 USER32.dll!gapfnScSendMessage +0x1ce (0x75fdc4e7 {USER32.dll+0x1c4e7})
so for each line in results.txt you have to parse out the offset and invoke symbolquery on the module like below
:\>symquery.exe -f -e c:\Windows\System32\user32.dll -a +0x1c4e7
InternalCallWinProc+0x23
??:0
:\>symquery.exe -f -e c:\Windows\System32\user32.dll -a +0x1c5e7
UserCallWinProcCheckWow+0xb3
a simple test processing example from a result.txt and a trimmed output
:\>grep "^#" results.txt | sed s/".*<"//g
# 0 system call NtUserBuildPropList parameter #2
USER32.dll+0x649d9>)
snip
COMCTL32.dll+0x2f443>)
notice the comctl32.dll (there is a default comctl.dll in system32.dll and several others in winsxs you have to consult the other files like global.log to view the dll load path
symquery.exe -f -e c:\Windows\winsxs\x86_microsoft.windows.common-
controls_6595b64144ccf1df_6.0.7601.17514_none_41e6975e2bd6f2b2\comctl32.dll -a +0x2f443
CallOriginalWndProc+0x1a
??:0
symquery.exe -f -e c:\Windows\system32\comctl32.dll -a +0x2f443
DrawInsert+0x120 <----- wrong symbol due to wrong module (late binding
/forwarded xxx yyy reasons)
I have a virtual Windows 7 x64 machine on a Windows 10 host, and I kernel debug it with windbg 10.0.10586.567. I'm running my own application on it, which I have full source and private symbols for. Whenever I break in and ask for stack traces of the app's threads, the backtrace always stops when one of my application's binaries are "hit."
So for instance, if I break in, switch to the process, and request a stacktrace with !thread [thread address] 1f, I get something like this (note the "early" zero return address at the last line):
fffff880`0534e870 fffff800`026d6992 nt!KiSwapContext+0x7a
fffff880`0534e9b0 fffff800`026d81a2 nt!KiCommitThreadWait+0x1d2
fffff880`0534ea40 fffff800`029c7a2e nt!KeDelayExecutionThread+0x186
fffff880`0534eab0 fffff800`026d08d3 nt!NtDelayExecution+0x59
fffff880`0534eae0 00000000`76e7165a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame # fffff880`0534eae0)
00000000`00276708 000007fe`fcf91203 ntdll!NtDelayExecution+0xa
00000000`00276710 00000001`410e7dd9 KERNELBASE!SleepEx+0xab
00000000`002767b0 00000000`00000000 MyApp!MainMessageLoop+0x4b1 [d:\whatever\path\myapplication.cpp # 3024]
This looks very similar to when you you are missing a binary while debugging a user-mode dump (lack of unwind data) of an x64 process, except in that case the stack trace usually does not stop "this sudden", rather it goes astray at that point, and shows bogus values.
Some extra info/things I tried:
I have the correct symbol paths set up (both the Microsoft symbol server, and a local folder on the host with matching PDBs, even though the latter is not needed for just the stack trace)
I have a binary path set up (.exepath) containing matching binaries on the host (I've made absolutely sure of this; copied the binaries directly from the guest to the host machine)
If I put a breakpoint in one of the app's exported DLL functions, then when the debugger breaks in, I get a one-liner stack trace like this: 0000000000274b40 0000000000000000 MyAppDLL!SomeExportedFunction+0x32 [d:\whatever\path\myapplicationDLL.cpp # 232]
I've tried virtually every combination of commands to get a stacktrace (.process /i, .process /r /p, !process -1 7, .reloads, .reload /users, .reload /f MyApp.exe, !thread [address] 1f, etc.) with no success
Tried with an older version of windbg (6.11.0001.404) as well, same result
Also tried on Windows 8.1 as a guest with the very same binaries, same result
!sym noisy output (irrelevant lines omitted):
0: kd>.process /i [address]
0: kd>g
0: kd>.reload /user
0: kd> !process -1 2
0: kd> !thread [address] 1f
[...]
DBGHELP: d:\symbolcache\MyApp.pdb\76931C5A6C284779AD2F916CA324617E1\MyApp.pdb already cached
DBGHELP: MyApp - private symbols & lines
[...]
lmvm MyApp output:
[...]
Loaded symbol image file: MyApp.exe
Image path: C:\MyApp\MyApp.exe
[...]
Any ideas?
I accidentally stumbled into a linker switch that solves this problem: /DEBUGTYPE with the PDATA argument. If you link your binaries with this switch, unwind information will be copied into your PDBs.
I recompiled/relinked the application in question with /DEBUGTYPE:CV,PDATA (/DEBUGTYPE:CV is the default if /DEBUG is specified, see the documentation), now everything works like a charm, I always get full call stacks.
One strange aspect of this: windbg happily uses unwind data found in the PDBs, but ignores the very same data in the mapped binaries (both on the host machine).
This is not a perfect solution to the problem (or any solution at all, one might say), but I'm providing this provisional answer with a workaround.
You should be able to get the information you want, albeit not so well-formatted using something like dps #rsp L10.
In x86-64 you don't have a parallel of the x86 ebp-chain, but the return addresses are still on the stack. Those will give you the functions in the stack, and the values between them will be the arguments passed to the functions (and saved registers on the stack, etc.). A random example from Google (as I'm not on my Windows machine right now):
0:017> dps #rsp
00000000`1bb0fbb8 00000000`00000020
00000000`1bb0fbc0 00000000`00000000
00000000`1bb0fbc8 00000000`008bc6c6 Dolphin!ReadDataFromFifoOnCPU+0xb6 [d:\sources\comex\source\core\videocommon\fifo.cpp # 245]
00000000`1bb0fbd0 00000000`1ba0ffeb
00000000`1bb0fbd8 00000000`00000020
00000000`1bb0fbe0 00000000`00000020
00000000`1bb0fbe8 00000000`00000800
00000000`1bb0fbf0 00000000`1ba0ffeb
00000000`1bb0fbf8 00000000`008c2ff5 Dolphin!InterpretDisplayListPreprocess+0x45 [d:\sources\comex\source\core\videocommon\opcodedecoding.cpp # 87]
00000000`1bb0fc00 00000000`00000000
00000000`1bb0fc08 00000000`008bc041 Dolphin!RunGpu+0x81 [d:\sources\comex\source\core\videocommon\fifo.cpp # 389]
00000000`1bb0fc10 00000000`8064cbc0
00000000`1bb0fc18 00000000`1bb0fcc0
00000000`1bb0fc20 00000000`00000000
00000000`1bb0fc28 00000000`008c2dda Dolphin!OpcodeDecoder_Preprocess+0x14a [d:\sources\comex\source\core\videocommon\opcodedecoding.cpp # 326]
00000000`1bb0fc30 00000000`8064cbe0
Given that you have symbols, the return addresses are easily distinguishable.
The unwind data is lazy loaded for user mode modules, so it's not going to be mapped unless someone needs it. Unfortunately the kernel debugger doesn't force the information to be present for user images, so sometimes you get this behavior. You can see if the data is mapped or not by dumping the PE header (!dh) and checking the state of the Exception Directory (!pte imagename+offset).
Given that you own the app, try forcing the information to be resident by doing a stack walk NOP somewhere in your app:
PVOID stack[2];
(VOID)CaptureStackBackTrace(0, 2, (PVOID*)&stack, NULL);
That doesn't guarantee the entire directory will be present, but usually good enough.