I am debugging a managed application using Son of Strike (SOS) in Visual studio 2010. I want to run a raw memory dump from a specific location, but I get "End of expression expected" error. If I attach WinDbg, then I can run same 'dd' command. How can I fix this problem?
!clrstack -l
OS Thread Id: 0xd5c (3420)
Child SP IP Call Site
0050eeac 002700eb ConsoleApplication2.Program.Main(System.String[])
LOCALS:
0x0050eeb0 = 0x0240c178
0x0050eebc = 0x00000000
0050f0fc 6b4c21bb [GCFrame: 0050f0fc]
dd 0x0240c178
End of expression expected
dd 0x0050eeb0
End of expression expected
In the Immediate window you have to use >dd 0x001AF2E0 to make it work. You have to type the > before dd.
dd 0x001AF2E0
End of expression expected
>dd 0x001AF2E0
0x001AF2E0 6d7c4938 ffffffff 001af34c 00000001
0x001AF2F0 002dd780 00000000 002dd780 ffffffff
0x001AF300 00000001 77a220f9 00000000 00713000
0x001AF310 002711a8 00000001 00000000 00000000
In the Command window you can just type dd 0x001AF2E0.
Type .cordll and see if the sos dll is loaded.
eg:
0:000> .cordll
CLR DLL status: Loaded DLL C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll
Related
All!
I'm debugging one quite strange case of process hanging/running out of memory using standard Windows crash dump with WinDbg. Obviously, it runs out of address space because of too many threads being created (it is 32 bit process), and I'm trying to figure out what's wrong with threads initialization (see callstack #3 below), because besides threads with callstacks that are typical for this program, it has handful of threads with callstacks of 3 types like:
1)
00 02cefb08 77544413 02fc024c 00000000 02cefb8c ntdll_774f0000!NtWaitForAlertByThreadId+0xc
01 02cefb28 7754434d 00000000 00000000 ffffffff ntdll_774f0000!RtlpWaitOnAddressWithTimeout+0x33
02 02cefb6c 7754423f 00000004 00000000 00000000 ntdll_774f0000!RtlpWaitOnAddress+0xa5
03 02cefba8 7752a605 02fc0000 02fc0000 02fc04b0 ntdll_774f0000!RtlpWaitOnCriticalSection+0xaa
04 02cefbc8 7752a525 02fc0248 02cefc88 77533844 ntdll_774f0000!RtlpEnterCriticalSectionContended+0xd5
05 02cefbd4 77533844 02fc0248 62da3da7 02fc04b0 ntdll_774f0000!RtlEnterCriticalSection+0x45
06 02cefc88 77533688 02fc04b0 02fc04b8 00000007 ntdll_774f0000!RtlpFreeHeap+0x174
07 02cefcd8 110d27fc 02fc0000 00000000 02fc04b8 ntdll_774f0000!RtlFreeHeap+0x758
...
These threads are stuck behind critical section 02fc024c that is taken by non-longer existing thread, and it is quite hard to figure out, what happened to it.
There are some threads that try to end normally, but are stuck in the LdrpDrainWorkQueue:
2)
# ChildEBP RetAddr Args to Child
00 05e5fd54 77527631 00000064 00000000 00000000 ntdll_774f0000!NtWaitForSingleObject+0xc
01 05e5fd78 7752b105 65f13f5f 00404e7c 00000000 ntdll_774f0000!LdrpDrainWorkQueue+0xbd
02 05e5fe70 7755179c 00404e7c 00404e7c 1086eb50 ntdll_774f0000!LdrShutdownThread+0x85
03 05e5ff40 00404efe 00000000 0042cef4 0042cefc ntdll_774f0000!RtlExitUserThread+0x4c
04 05e5ff6c 00404ea6 05e5ffcc 004049b8 05e5ff80 abc!EndThread+0x6
05 05e5ff80 743962c4 1086eb50 743962a0 941a355e abc!ThreadWrapper+0x2a
06 05e5ff94 77550779 1086eb50 65f13ef3 00000000 kernel32!BaseThreadInitThunk+0x24
07 05e5ffdc 77550744 ffffffff 77573606 00000000 ntdll_774f0000!__RtlUserThreadStart+0x2f
08 05e5ffec 00000000 00404e7c 1086eb50 00000000 ntdll_774f0000!_RtlUserThreadStart+0x1b
Also, dump presents about 1400 threads on a very early stage of initialization, that were created during last 5 minutes of process life with a callstack like:
3)
# ChildEBP RetAddr Args to Child
00 0ed1fba4 77527631 00000064 00000000 00000000 ntdll_774f0000!NtWaitForSingleObject+0xc
01 0ed1fbcc 7752b586 6ec53d9b ffffffff 1ed9d000 ntdll_774f0000!LdrpDrainWorkQueue+0xbd
02 0ed1fcb4 77557d86 6ec53c27 00000000 00000000 ntdll_774f0000!LdrpInitializeThread+0x8d
03 0ed1fd08 77557ce0 00000000 00000000 0ed1fd24 ntdll_774f0000!_LdrpInitialize+0x6a
04 0ed1fd10 00000000 0ed1fd24 774f0000 00000000 ntdll_774f0000!LdrInitializeThunk+0x10
These threads are also waiting in the LdrpDrainWorkQueue for event LdrpLoadCompleteEvent to be signalled.
This event is related to parallel loader (for some reference, first answer fot his question from RbMm, somewhat similar yet different situation here) This event is created during process initialization and signalled after parallel DLL loading has finished, so all LdrpInitializeThread's could traverse DllMain's and signal THREAD_ATTACH. But I don't understand why it is in non-signalled state on a process that has been running for weeks? Does parallel loader work on LoadLibrary as well, so LdrpLoadCompleteEvent gets reset? Couldn't find it in disassembly.
In any case, I'm trying to understand why process has developed such strange callstacks before it was forcefully terminated. I could imagine, that some thread began loading DLL that caused LdrpLoadCompleteEvent to be reset, then some thread holding lock for the heap died in a bad manner, so dll loading couldn't have been completed, so LdrpLoadCompleteEvent was never signalled, hence no new threads could have been initialized. However, there's no any thread that is loading dll in the dump.
Any insight/hint regarding how such callstacks could have been developed, or what else I could do to squeeze more info from the dump, is welcome.
Thank you!
Your fundamental problem is architectural ... a legendary problem known as "thrashing."
Your system is probably designed with what I refer to as the "flaming-arrow approach." Whenever a new request comes in, "just light another flaming arrow and throw it into the air." Unfortunately you just can't do that.
The permanent solution to your problem unfortunately will never be solved by a debugger: it will require a redesign.
I am familiar with using windbg or IDA for remote kernel debugging, but right now i have extracted a kernel driver from an executable, and have done static analysis on its IDB and renamed a lot of variables, what is the easiest way of using my IDB file to debug the driver on the remote debugee when it gets loaded by the executable?
I know how to attach to remote kernel using IDA, but how can i use my current IDB file, and put breakpoint on some of its functions so it they get hit when the driver is loaded? (I dont have the corresponding pdb file for the driver so i can't use symbols for breakpoint)
this is a vanilla windbg answer to break on any DriverInit
once you have Broken on DriverInit You Can Lookup and Set bp on all MajorFunctions
Assuming you have a regular kd Connection
use sxe -ibp;.reboot to reboot the target
on reconnection the target will break very Early as below
kd> sxe ibp;.reboot
Shutdown occurred at (Sun Oct 18 02:58:09.077 2020 )...unloading all symbol tables.
Waiting to reconnect...
Connected to Windows 7 7601 x86 compatible target at (xxx), ptr64 FALSE
Kernel Debugger connection established. (Initial Breakpoint requested)
once broken set a breakpoint on nt!IopLoadDriver
inside this function search for an indirect Call
that Calls the _DRIVER_OBJECT->DriverInit
kd> ?? #FIELD_OFFSET(nt!_DRIVER_OBJECT , DriverInit)
long 0x2c
like
nt!IopLoadDriver+0x7ea:
829d5355 ff562c call dword ptr[esi+2Ch] ds:84f2928c={cdrom!FxDriverEntry (87eb53cf)}
set a break point here to
you are now set to enter almost every driver that is loaded
once you are on entrypoint of any Driver
use the DriverObject (an argument the DriverEntry Takes )
and Set Breakpoints on each MAJORFunction
kd> bp . "du poi(#esi+1c+4);gc"
kd> bl
0 e Disable Clear 829d4b6a 0001 (0001) nt!IopLoadDriver
1 e Disable Clear 829d5355 0001 (0001) nt!IopLoadDriver+0x7ea "du poi(#esi+1c+4);gc"
kd> bd 0
kd> bl
0 d Enable Clear 829d4b6a 0001 (0001) nt!IopLoadDriver
1 e Disable Clear 829d5355 0001 (0001) nt!IopLoadDriver+0x7ea "du poi(#esi+1c+4);gc"
kd> g
841bd1d0 "\Driver\Null.Ѕ捁印䍁䥐停偎〰〰"
84f18718 "\Driver\Beep.Б浍摌䂈蓶䈸蓶...."
84eef210 "\Driver\VgaSave"
84eb2860 "\Driver\RDPCDDᛛ..В浍慃憠褎.蓫菌蓲"
84e903c0 "\Driver\RDPENCDD..浍摌읨蓤潤獷獜獹整.尲牤癩牥"
84e90400 "屳摲数据摤献獹"
84ef15c0 "\Driver\RDPREFMP..牉..蓧"
84ef4a78 "\FileSystem\Msfs.В浍慃冀褘蝴蓳荤蓶"
84f191f0 "\FileSystem\Npfs.З獍䑆.°"
I have a single DLL or EXE file and I'd like to acquire the symbols (PDB file) for it, i.e. download from the symbol server.
Note: The DLL is not a module which is part of a crash dump, so opening the dump and doing .symfix and .reload is not possible.
I have tried
symchk /os /if "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll"
/su "SRV*e:\debug\symbols*http://msdl.microsoft.com/download/symbols"
but it just says
SYMCHK: FAILED files = 0
SYMCHK: PASSED + IGNORED files = 1
and there are no symbols in my symbol folder.
You have used /if which sounds plausible, but actually it needs to be /id for a dump file:
symchk /os /id "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll"
/su "SRV*e:\debug\symbols*http://msdl.microsoft.com/download/symbols"
SYMCHK: FAILED files = 0
SYMCHK: PASSED + IGNORED files = 1
The output is the same, but the symbol folder contains the PDBs now.
It is similar in WinDbg:
choose File | Open Crash Dump ... or press Ctrl+D
for the file name filter, instead of Crash Dump Files select All files
choose the DLL or EXE of your interest. WinDbg will e.g. say
Loading Dump File [C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll]
although it is not really a dump file
issue the typical commands .symfix and .reload. If symbols are present on the symbol server, they will be downloaded.
Looking at what you're "debugging", you'll see that it's the DLL:
0:000> |
. 0 id: f0f0f0f0 examine name: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
0:000> ||
. 0 Image file: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
0:000> lm
start end module name
00000001`80000000 00000001`80988000 clr (pdb symbols) e:\debug\symbols\clr.pdb\5706A2AA257A45FDAC5776EDDC7BBA542\clr.pdb
And also some other commands work:
0:000> x clr!*
00000001`80123e28 clr!SafeHandle::Init (<no parameter info>)
00000001`808f5e80 clr!HillClimbingLogSize = <no type information>
00000001`80064af0 clr!IsTimerSpecialThread (<no parameter info>)
...
0:000> u clr!SafeHandle::Init
clr!SafeHandle::Init:
00000001`80123e28 4883ec28 sub rsp,28h
00000001`80123e2c 488b059d4b7c00 mov rax,qword ptr [clr!g_Mscorlib+0x10 (00000001`808e89d0)]
00000001`80123e33 488b80e0070000 mov rax,qword ptr [rax+7E0h]
00000001`80123e3a 4885c0 test rax,rax
...
I looked for this question over the site but it looks like it's never been asked, so I thought I would share the answer to anyone looking for it.
Q: When I input executables files' name in cmd.exe, through shell-execution functions or from a Batch file, how does Windows locate the external executable file to be called?
I do not understand what "shell-execution function" is referring to, so I will ignore it.
But the rules for launching an executable from the CMD.EXE command line, or from a batch file are as follows:
If a path is specified for the executable (absolute or relative), then only that path is searched.
If only the executable name (and possibly extension) is specified, then
First search the current folder
Then search folders in PATH environment variable (in order listed)
Within each folder searched, use the provided extension. If no file extension is provided, then look for files that match extensions found in PATHEXT environment variable (in order listed).
The first matching file found is the one that gets executed.
EDIT
David Candy has pointed me to some outdated NT documentation that precisely outlines the search process, and it essentially matches what I have written above. See Launching Applications from the NT Shell
I'd love it if anyone could post links to similar CMD.EXE documentation for more recent versions of Windows, especially if it was written by Microsoft. Note that the NT doc linked above was not written by Microsoft, even though it is posted on their site.
Here's a debug of CMD using CreateProcessW to start edit.com.
Microsoft (R) Windows Debugger Version 6.2.9200.20512 X86
Copyright (c) Microsoft Corporation. All rights reserved.
CommandLine: C:\Windows\System32\cmd.exe
Symbol search path is: srv*C:\tmp*http://msdl.microsoft.com/download/symbols;c:\tmp
Executable search path is:
ModLoad: 49fe0000 4a030000 cmd.exe
ModLoad: 76dd0000 76ef7000 ntdll.dll
ModLoad: 75860000 7593b000 C:\Windows\system32\kernel32.dll
ModLoad: 76a20000 76ae6000 C:\Windows\system32\ADVAPI32.dll
ModLoad: 76d00000 76dc2000 C:\Windows\system32\RPCRT4.dll
ModLoad: 75b20000 75bca000 C:\Windows\system32\msvcrt.dll
(58c.918): Break instruction exception - code 80000003 (first chance)
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll -
eax=00000000 ebx=00000000 ecx=0027f9a0 edx=76e29a94 esi=fffffffe edi=76e2b6f8
eip=76e17dfe esp=0027f9b8 ebp=0027f9e8 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
ntdll!DbgBreakPoint:
76e17dfe cc int 3
0:000> bp kernel32!CreateProcessW
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Windows\system32\kernel32.dll -
breakpoint 0 redefined
0:000> g
Breakpoint 0 hit
eax=0027f590 ebx=00000000 ecx=00000000 edx=00000000 esi=4a005200 edi=00000001
eip=75861c01 esp=0027f4b4 ebp=0027f674 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
kernel32!CreateProcessW:
75861c01 8bff mov edi,edi
0:000> kb
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
0027f4b0 49fe4568 00316078 003175a8 00000000 kernel32!CreateProcessW
0027f674 49fe4315 00317508 00000000 00000000 cmd!ExecPgm+0x20a
0027f8d4 49fe43a8 00317508 00000000 00000000 cmd!ECWork+0x7f
0027f8ec 49fe43ce 00317508 e3a3ec47 00000001 cmd!ExtCom+0x47
0027fd48 49fe185f 00317508 00000002 7588372f cmd!FindFixAndRun+0xb3
0027fd98 49ff70d4 00000000 00317508 4a0041b0 cmd!Dispatch+0x14a
0027fddc 49fe985b 00000001 000a0f38 000a1840 cmd!main+0x21a
0027fe20 758a4911 7ffdb000 0027fe6c 76e0e4b6 cmd!_initterm_e+0x163
0027fe2c 76e0e4b6 7ffdb000 7871d1d4 00000000 kernel32!BaseThreadInitThunk+0x12
0027fe6c 76e0e489 49fe9797 7ffdb000 00000000 ntdll!RtlInitializeExceptionChain+0x63
0027fe84 00000000 49fe9797 7ffdb000 00000000 ntdll!RtlInitializeExceptionChain+0x36
0:000> du 316078
00316078 "C:\Windows\system32\edit.com"
See CreateProcess.
The directory from which the application loaded.
The current directory for the parent process.
The 32-bit Windows system directory. Use the GetSystemDirectory function to get the path of this directory.
Windows Me/98/95: The Windows system directory. Use the GetSystemDirectory function to get the path of this directory.
The 16-bit Windows system directory. There is no function that obtains the path of this directory, but it is searched. The name of this directory is System.
The Windows directory. Use the GetWindowsDirectory function to get the path of this directory.
The directories that are listed in the PATH environment variable. Note that this function does not search the per-application path specified by the App Paths registry key. To include this per-application path in the search sequence, use the ShellExecute function.
The question may be a bit awkward, but here's my detailed problem:
Currently I'm looking into setting up SysInternals' procdump.exe to monitor an application of ours that exhibits spurious disappearances -- that is, the user reports that the application is simply "gone" without any trace after a short visible hang of the application's window.
My first idea was to run procdump -e -x . MyApp.exe which would record a crash dump when the application encounters an unhandled exception, but then I saw that there is also a -t switch, that --
-t - Write a dump when the process terminates.
automatically generates a dump when the process terminates.
Now the problem
I have tested the -t switch with our app by inserting a ExitProcess or TerminateProcess call at a defined location where I can trigger it.
While the app behaves as expected, i.e. TerminateProcess immediately "kills" the running app and ExitProcess takes a while because global cleanup is run, the dump generated this way is useless in both cases.
The dumps I get for -t always contain only a sinlge thread (where the app was running over 20 thread at termination time) and the callstack isn't even at a useful location. (It just seems to be one random thread from the terminated app.)
Am I doing something wrong? Can I usefully use procdump -t to track down unexpected calls of process exit functions at all?
Can I usefully use procdump -t to track down unexpected calls of
process exit functions at all?
I think not and here's why:
test process calc.exe
CommandLine: "C:\Program Files\Sysinternals\procdump.exe" -t calc.exe
I try to carefully suggest that procdump is waiting on calc.exe process handle.
0:000> kb
ChildEBP RetAddr Args to Child
0017f2e0 77135e6c 75336872 00000002 0017f334 ntdll!KiFastSystemCallRet
0017f2e4 75336872 00000002 0017f334 00000001 ntdll!NtWaitForMultipleObjects+0xc
0017f380 76cbf14a 0017f334 0017f3a8 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100
0017f3c8 76cbf2c2 00000002 7ffdb000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0
0017f3e4 011c6135 00000002 0017f46c 00000000 kernel32!WaitForMultipleObjects+0x18
WARNING: Stack unwind information not available. Following frames may be wrong.
0017fc30 011c999e 00000003 013d1de0 013d1e78 procdump+0x6135
0017fc78 76cc1194 7ffdb000 0017fcc4 7714b495 procdump+0x999e
0017fc84 7714b495 7ffdb000 77ad79b5 00000000 kernel32!BaseThreadInitThunk+0xe
0017fcc4 7714b468 011c99f5 7ffdb000 00000000 ntdll!__RtlUserThreadStart+0x70
0017fcdc 00000000 011c99f5 7ffdb000 00000000 ntdll!_RtlUserThreadStart+0x1b
0:000> dd 17f46c
0017f46c 00000238 00000268
0:000> !handle 238 f
Handle 238
Type Process
Attributes 0
GrantedAccess 0x1fffff:
Delete,ReadControl,WriteDac,WriteOwner,Synch
Terminate,CreateThread,,VMOp,VMRead,VMWrite,DupHandle,CreateProcess,SetQuota,SetInfo,QueryInfo,SetPort
HandleCount 5
PointerCount 52
Name <none>
Object Specific Information
Process Id 1580
Parent Process 2476
Base Priority 8
In the crash dump file gets stack last complete process thread (TID 3136) just before the end of the process.
0:000> ~
. 0 Id: dc8.c40 Suspend: -1 Teb: 7ffdd000 Unfrozen
0:000> .formats c40
Evaluate expression:
Hex: 00000c40
Decimal: 3136
Crash dump file is created after the completion of the last thread, and before the end of the process.