I'm trying to debug an issue with what may be handle leak. I have a dump created on a remote windows machine and I would like to see the handles information. I'm using WinDbg. I have seen some articles from the MSDN and from other sources, like https://www.codeproject.com/Articles/6988/Debug-Tutorial-Part-5-Handle-Leaks, but I can't get it to work, so I need some help.
I tried the next
Using !handles or !devhandles - these have failed. I either get no export handles found or a failure to load kdexts. Apparently kernel debugging is not enabled.
I found winxp/kdexts.dll in my path (given from .chain command) but it wouldn't load - .load kdexts yields `DebugExtensionInitializeFailed1 with error code 0x80004005.
Using !handle - with !handle -? I get help for the command but when I try something else, I get "Unable to read handle information". For example,
!handle - I expected a full list of handles
!handle 0 0
!handle 0 0 file
My setup
The remote process is Windows server 2012 (64bit), just as my own machine
I'm using the latest WinDbg from the windows sdk 10
I have a full dump, created by right-clicking task manager
I need some help if possible
Do I need to do kernel-debugging in order to view the list of handles from the dump?
Is it possible at all to do kernel-debugging on a full dump created from task-manager? Or is it required that the dump be taken differently?
How can I know if a given dump file includes the handle information?
How can I use the !handle command properly?
Is there any other alternative, such as using visual studio, another utility, etc.?
I'd appreciate any help
!tamir
your dump was probably a dump taken without handle information
you may use dumpchk.exe that comes with windbg installation to see if Handle Stream exists in the dump
if you have control over dump creation check how to use .dump /ma with windbg
or you may also explore sysinternals procdump.exe
and also make sure you are using the correct bitted debugger for the dump in question
a sample path
D:\>dir /s /b "c:\Program Files (x86)\Windows Kits\10\Debuggers\cdb.exe"
c:\Program Files (x86)\Windows Kits\10\Debuggers\arm\cdb.exe
c:\Program Files (x86)\Windows Kits\10\Debuggers\arm64\cdb.exe
c:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe
c:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe
here is a sample dump creation with and without handle stream in the dump
:000> .dump /ma d:\madump.dmp
Creating d:\madump.dmp - mini user dump
Dump successfully written
0:000> .dump d:\nomadump.dmp
Creating d:\nomadump.dmp - mini user dump
Dump successfully written
0:000> q
analysing both the dumps with dumpchk and checking for streams present
dumpchk nomadump.dmp > nomachk.txt
dumpchk madump.dmp > machk.txt
D:\>type machk.txt |grep -i number.*stream
NumberOfStreams 17
D:\>type nomachk.txt |grep -i number.*stream
NumberOfStreams 13
diff
D:\>diff -y machk.txt nomachk.txt
Microsoft (R) Windows Debugger Version 10.0.17763.132 AMD64 Microsoft (R) Windows Debugger Version 10.0.17763.132 AMD64
Loading Dump File [D:\madump.dmp] | Loading Dump File [D:\nomadump.dmp]
User Mini Dump File with Full Memory: Only application d | User Mini Dump File: Only registers, stack and portions of me
----- User Mini Dump Analysis ----- User Mini Dump Analysis
MINIDUMP_HEADER: MINIDUMP_HEADER:
Version A793 (A063) Version A793 (A063)
NumberOfStreams 17 | NumberOfStreams 13
Flags 441826 | Flags 40000
0002 MiniDumpWithFullMemory <
0004 MiniDumpWithHandleData <
0020 MiniDumpWithUnloadedModules <
0800 MiniDumpWithFullMemoryInfo <
1000 MiniDumpWithThreadInfo <
40000 MiniDumpWithTokenInformation 40000 MiniDumpWithTokenInformation
400000 MiniDumpWithIptTrace <
if you feel enterprising take a look here for some hints to deciphering a dump without windbg /dbgeng
forgot to post the result of doing !handle on both dumps
D:\>cdb -c "!handle;q" -z nomadump.dmp |awk /Reading/,/quit/"
0:000> cdb: Reading initial command '!handle;q'
ERROR: !handle: extension exception 0x80004002.
"Unable to read handle information"
quit:
D:\>cdb -c "!handle;q" -z madump.dmp |awk /Reading/,/quit/"
0:000> cdb: Reading initial command '!handle;q'
Handle 0000000000000004
Type File
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxSNIPxxxxxxxxx
Handle 0000000000000128
Type Mutant
Handle 000000000000012c
Type
Handle 0000000000000180
Type File
70 Handles
Type Count
None 27
Event 13
File 8
Directory 2
Mutant 1
Semaphore 2
Key 6
IoCompletion 2
TpWorkerFactory 2
ALPC Port 1
WaitCompletionPacket 6
quit:
Check the tool which was used for creating the crash dump. Perhaps it provides an option to include handle data.
Task Manager includes handle data by default
Visual Studio includes handle data by default
In WinDbg, .dump can be used with the /mh switch to include handle data. /ma is a shortcut for /mfFhut, so it also includes handle data.
ProcDump automatically includes handle data.
Windows Error Reporting LocalDumps can be configured with a Registry value called CustomDumpFlags.
If you create the dump programmatically yourself with MiniDumpWriteDump(), use MINIDUMP_TYPE::MiniDumpWithHandleData.
Related
After running a set of tests with drmemory overnight I am trying to resolve the error stacks by providing pdb symbols. The pdb's come from a large samba-mapped repository and using _NT_SYMBOL_PATH at runtime slowed things down too much.
Does anyone know of a tool that post-processes results.txt and pulls new symbols (via NT_SYMBOL_PATH or otherwise) as required to produce more detailed stacks ? If not, any hints for adapting asan_symbolize.py to do this ?
https://llvm.org/svn/llvm-project/compiler-rt/trunk/lib/asan/scripts/asan_symbolize.py
What I came up with so far using dbghelp.dll is below. Works but could be better.
https://github.com/patraulea/postpdb
ok this Query does not pertain to use of windbg or doesn't have anything to do with _NT_SYMBOL_PATH
Dr.Memory is a memory diagnostic tool akin to valgrind and is based on Dynamorio instumentation framework usable on raw unmodified binaries
on windows you can invoke it like drmemory.exe calc.exe from a command prompt (cmd.exe)
as soon as the binary finishes execution a log file named results.txt is written to a default location
if you had setup _NT_SYMBOL_PATH drmemory honors it and resolves symbol information from prepulled symbol file (viz *.pdb) it does not seem to download files from ms symbol server it simply seems to ignore the SRV* cache and seems to use only the downstream symbol folder
so if the pdb file is missing or isnt downloaded yet
the results.txt will contain stack trace like
# 6 USER32.dll!gapfnScSendMessage +0x1ce (0x75fdc4e7 <USER32.dll+0x1c4e7>)
# 7 USER32.dll!gapfnScSendMessage +0x2ce (0x75fdc5e7 <USER32.dll+0x1c5e7>)
while if the symbol file was available it would show
# 6 USER32.dll!InternalCallWinProc
# 7 USER32.dll!UserCallWinProcCheckWow
so basically you need the symbol file for appplication in question
so as i commented you need to fetch the symbols for the exe in question
you can use symchk on a running process too and create a manifest file
and you can use symchk on a machine that is connected to internet
to download symbols and copy it to a local folder on a non_internet machine
and point _NT_SYMBOL_PATH to this folder
>tlist | grep calc.exe
1772 calc.exe Calculator
>symchk /om calcsyms.txt /ip 1772
SYMCHK: GdiPlus.dll FAILED - MicrosoftWindowsGdiPlus-
1.1.7601.17514-gdiplus.pdb mismatched or not found
SYMCHK: FAILED files = 1
SYMCHK: PASSED + IGNORED files = 27
>head -n 4 calcsyms.txt
calc.pdb,971D2945E998438C847643A9DB39C88E2,1
calc.exe,4ce7979dc0000,1
ntdll.pdb,120028FA453F4CD5A6A404EC37396A582,1
ntdll.dll,4ce7b96e13c000,1
>tail -n 4 calcsyms.txt
CLBCatQ.pdb,00A720C79BAC402295B6EBDC147257182,1
clbcatq.dll,4a5bd9b183000,1
oleacc.pdb,67620D076A2E43C5A18ECD5AF77AADBE2,1
oleacc.dll,4a5bdac83c000,1
so assuming you have fetched the symbols it would be easier to rerun the tests with a locally cached copies of the symbol files
if you have fetched the symbols but you cannot rerun the tests and have to work solely with the output from results.txt you have some text processing work (sed . grep , awk . or custom parser)
the drmemory suite comes with a symbolquery.exe in the bin folder and it can be used to resolve the symbols from results.txt
in the example above you can notice the offset relative to modulebase like
0x1c4e7 in the line # 6 USER32.dll!gapfnScSendMessage +0x1ce (0x75fdc4e7 {USER32.dll+0x1c4e7})
so for each line in results.txt you have to parse out the offset and invoke symbolquery on the module like below
:\>symquery.exe -f -e c:\Windows\System32\user32.dll -a +0x1c4e7
InternalCallWinProc+0x23
??:0
:\>symquery.exe -f -e c:\Windows\System32\user32.dll -a +0x1c5e7
UserCallWinProcCheckWow+0xb3
a simple test processing example from a result.txt and a trimmed output
:\>grep "^#" results.txt | sed s/".*<"//g
# 0 system call NtUserBuildPropList parameter #2
USER32.dll+0x649d9>)
snip
COMCTL32.dll+0x2f443>)
notice the comctl32.dll (there is a default comctl.dll in system32.dll and several others in winsxs you have to consult the other files like global.log to view the dll load path
symquery.exe -f -e c:\Windows\winsxs\x86_microsoft.windows.common-
controls_6595b64144ccf1df_6.0.7601.17514_none_41e6975e2bd6f2b2\comctl32.dll -a +0x2f443
CallOriginalWndProc+0x1a
??:0
symquery.exe -f -e c:\Windows\system32\comctl32.dll -a +0x2f443
DrawInsert+0x120 <----- wrong symbol due to wrong module (late binding
/forwarded xxx yyy reasons)
I want to create a batch file which runs a program for 7 seconds, irrespective of the completion of execution of the other program. I also want that the program should take input and save the output in an external file. This is what I tried:
start program.exe
ping 1.1.1.1 -n 1 -w 7000 > nul
taskkill /im program.exe /f
rem continue here
The above works fine, but when I replace line 1 with:
start program.exe < in.txt > out.txt
then input from file and output in file doesn't work.
cmd doesn't set the STARTF_USESTDHANDLES flag of the CreateProcess STARTUPINFO structure. Instead, it temporarily redirects its own standard handles and relies on inheritance. This approach works even if cmd has to call ShellExecuteEx, which lacks a way to explicitly set the standard handles.
However, redirecting its own standard handles doesn't work when CREATE_NEW_CONSOLE is set in the process creation flags, which is the default for the start command. To avoid this problem, use the /b option to prevent creating a new console.
You may also want to redirect stderr to stdout or a file. This prevents errors from being written to the console. For example:
start /b program.exe <in.txt >out.txt 2>&1
start /b program.exe <in.txt >out.txt 2>err.txt
start /b program.exe <in.txt >out.txt 2>nul
Example using Debugging Tools for Windows
(test) C:\>cdb -Goxi ld cmd
Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
CommandLine: cmd
Symbol search path is: symsrv*symsrv.dll*
C:\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
(ed0.1770): Break instruction exception - code 80000003 (first chance)
ntdll!LdrpDoDebuggerBreak+0x30:
00000000`77848700 cc int 3
0:000> .reload /f
Reloading current modules
.....
0:000> bp CreateProcessW
0:000> g
Run where.exe in a new console.
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
(test) C:\>start /w where.exe <nul >nul
Breakpoint 0 hit
kernel32!CreateProcessW:
00000000`775a0660 4883ec68 sub rsp,68h
Note that cmd.exe redirects its StandardOutput before calling CreateProcess:
0:000> ?? ((ntdll!_PEB *)#$peb)->ProcessParameters->StandardOutput
void * 0x00000000`00000060
0:000> !handle 60 3
Handle 60
Type File
Attributes 0
GrantedAccess 0x120196:
ReadControl,Synch
Write/Add,Append/SubDir/CreatePipe,WriteEA,ReadAttr,WriteAttr
HandleCount 2
PointerCount 3
The process creation flags, i.e. dwCreationFlags, the 6th parameter:
0:000> dd (#rsp + 6*8) l1
00000000`00182c58 00080410
is passed as 0x80410, which is a bitwise OR of the following flags:
EXTENDED_STARTUPINFO_PRESENT
CREATE_UNICODE_ENVIRONMENT
CREATE_NEW_CONSOLE
Because a new console is created, where.exe doesn't inherit cmd's standard handles:
0:000> g
Symbol search path is: symsrv*symsrv.dll*
C:\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
(1550.1a80): Break instruction exception - code 80000003 (first chance)
ntdll!LdrpDoDebuggerBreak+0x30:
00000000`77848700 cc int 3
1:001> ?? ((ntdll!_PEB *)#$peb)->ProcessParameters->StandardOutput
void * 0x00000000`00000007
Note: in Windows 8+ a console handle is just a regular file handle, so you'll have to look deeper.
I'm using Windows 7 for this example, so console handles are fake handles tagged by setting the lower 2 bits (e.g. 3, 7, 11 => 0b0011, 0b0111, 0b1011). By 'fake' I mean they're not in the process handle table used for kernel object handles. Thus, for example, you can't use the debugger !handle command to inspect handle 7:
1:001> !handle 7 f
Could not duplicate handle 7, error 87
In Windows 7, console handles are allocated and managed by the console host process, conhost.exe. They're tagged so Windows base functions can make the required LPC call to conhost.exe via NtRequestWaitReplyPort.
The above example demonstrated how creating a new console overrides inheriting cmd's redirected standard handles. Now let's add the /b option to prevent creating a new console.
1:001> g
(test) C:\>start /b /w where.exe <nul >nul
Breakpoint 0 hit
kernel32!CreateProcessW:
00000000`775a0660 4883ec68 sub rsp,68h
dwCreationFlags is 0x80600:
0:000> dd (#rsp + 6*8) l1
00000000`00182c58 00080600
which is a bitwise OR of the following creation flags:
EXTENDED_STARTUPINFO_PRESENT
CREATE_UNICODE_ENVIRONMENT
CREATE_NEW_PROCESS_GROUP
(A side effect of specifying /b is to create the process as the leader of a new process group. If it's a console process, this allows generating a Ctrl+Break event that targets the group.)
In this case, where.exe does inherit the redirected standard handles from cmd.exe:
0:000> g
Symbol search path is: symsrv*symsrv.dll*
C:\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
(1508.1534): Break instruction exception - code 80000003 (first chance)
ntdll!LdrpDoDebuggerBreak+0x30:
00000000`77848700 cc int 3
1:001> ?? ((ntdll!_PEB *)#$peb)->ProcessParameters->StandardOutput
void * 0x00000000`00000064
1:001> !handle 64 3
Handle 64
Type File
Attributes 0
GrantedAccess 0x120196:
ReadControl,Synch
Write/Add,Append/SubDir/CreatePipe,WriteEA,ReadAttr,WriteAttr
HandleCount 3
PointerCount 4
Again, in Windows 7 it's easy to spot a console pseudo handle because it's tagged by setting the low 2 bits of the handle value. For Windows 8+, a quick check is to look at the low nibble (4 bits) of the file's granted access, for which read data access is 1, write data access is 2, and append data access is 4. A file opened for a console buffer has both read and write access, whereas cmd's redirection uses either read (<) or write (>), but not both. The above is the redirected output, and you can see the file is opened with write and append access (2+4), but not read access. That's a quick check, but if you want to be certain you can use a kernel debugger such as kd.exe, or a tool such as Sysinternals Process Explorer or handle.exe. These can show you the NT kernel object path, such as \Device\ConDrv\Input for a Windows 8+ console input handle.
Here is my registry:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps]
"DumpType"=dword:00000000
"CustomDumpFlags"=dword:00000006
According to this article DumpType=0 means custom dump and then CustomDumpFlags is taken into account. According to this article CustomDumpFlags=6 means MiniDumpWithFullMemory | MiniDumpWithHandleData, where:
MiniDumpWithFullMemory - Include all accessible memory in the process. The raw memory data is included at the end, so that the initial structures can be mapped directly without the raw memory information. This option can result in a very large file.
MiniDumpWithHandleData - Include high-level information about the operating system handles that are active when the minidump is made.
Now I have a crash-me application, so I run it, it crashes, the dump is created in %userprofile%\AppData\Local\CrashDumps, I open it in windbg and see the following line there:
User Mini Dump File with Full Memory: Only application data is
available
Which is equivalent to CustomDataFlags=2
So, how am I expected to create a dump with the handle data in it? If possible, I would like to use no third parties.
My OS is Windows 8 or Windows 2008R2 server or higher.
Try .dumpdebug, which is an undocumented command. At the top of the output there should be the flags:
0:006> .dumpdebug
----- User Mini Dump Analysis
MINIDUMP_HEADER: Version A793 (62F0) NumberOfStreams 15 Flags 41826
0002 MiniDumpWithFullMemory
0004 MiniDumpWithHandleData
0020 MiniDumpWithUnloadedModules
0800 MiniDumpWithFullMemoryInfo
1000 MiniDumpWithThreadInfo
40000 MiniDumpWithTokenInformation
If you dislike the verbose output, you can filter it with a find command on the shell:
.shell -ci ".dumpdebug" find "MiniDumpWith"
The following message is copied from WinDBG's output window:
Use !analyze -v to get detailed debugging information.
BugCheck 24, {1904fb, 8f9ec9f8, 8f9ec5d0, 87c4fccc}
Probably caused by : Ntfs.sys ( Ntfs!NtfsCommonRead+6f4 )
Followup: MachineOwner
Please note the last line: "Followup: MachineOwner"
What does it mean?
The Followup output is the attempt by the !analyze command to triage the exception to identify who owns the fault, sometimes it could be a driver, hardware, application etc..
There is a default entry in the triage.ini file that !analyze (and also !owner command) uses when it cannot find a match:
default=MachineOwner
This file is located in a folder called triage on my machine it is in
C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\triage
It is something you can ignore in your case, there is more information about how the analysis works and how to edit the triage.ini file for your specific use case.
I try to use logman instead of DDK tracelog for colecting *.etl data produced by my application that uses WPP but was not able to see any data (in *.etl) after reading the etl file and decoding *.fmt information by using traceview.
What am I doing wrong? I generate *etl like this
logman start "Session" -o "Trace.etl" -p "{28EE579B-CF67-43b6-9D19-8930E7AAA131}" -ets
logman stop "Session" -ets
When opening with traceview the generated Trace.etl it shows not errors only that there is no collected data there.
EDIT: I specify that I registered my generated *.mof file using mofcompiler on the system and that by using traceview directly I can see data.
The problem was the fact that I didn't specify any flags and because of that (WPP messages being with flags set I was not seeing any data:
logman start "Session" -o "Trace.etl" -p "{28EE579B-CF67-43b6-9D19-8930E7AAA131}" 0xFFFF -ets logman stop "Session" -ets
I was searching for the same issue and found this useful documentation on MSDN:
CLR ETW Keywords and Levels
The levels have the following meanings:
0x5 - Verbose
0x4 - Informational
0x3 - Warning
0x2 - Error
0x1 - Critical
0x0 - LogAlways