How do you debug a deadlocked Windows app on a customer machine - windows

I have a multi-threaded Windows application that occasionally deadlocks. Inevitably this happens on a customer system and not when the software is in test. What is the easiest way of getting a Windows Minidump of the current state of the application? If it could also terminate the application so the user can restart it and continue using the system that would be great.

In Vista you can create a dump file directly from task manager. Right click on a process in the processes tab and choose 'create dump file'.
Prior to Vista I prefer the ntsd route, since although it is not totally user friendly it works without the user installing any other software and the instructions are actually fairly easy to follow.

See the Microsoft support article How to use ADPlus to troubleshoot "hangs" and "crashes", as well as the helpful blog post Debugging Production Applications using ADPlus.
Both of these articles are about "ADPlus", a VBScript tool supplied with Debugging Tools for Windows that can be used to generate minidumps from a production environment (which can later be loaded up with WinDbg on your development machine). ADPlus has a lot of functionality and a lot of options, so it may take some reading, experimentation, and practice to find the best way to use it in your environment.

I know how to achieve this. It's just my technique is a bit clunky. All Windows 2000 and later systems have a basic command line debugger as part of the default install called NTSD. What I do at the moment is run:
ntsd -pn MyApp.exe
When the debugger console appears I can then type the following into the debugger console:
.dump c:\my-deadlock.mdmp
.kill
What I'm looking for is something that's a little bit cleaner and easier to put in an email to customers to just run. I've seen it alluded to somewhere (that google can't find for now) that you can use drwtsn32.exe to extract a crash dump and terminate an application.
Edit: It is possible to streamline the command somewhat:
ntsd -pn MyApp.exe -c ".dump c:\my-deadlock.mdmp; .kill"
The command .detach can be given if the process has not terminally hung (e.g. a long network timeout) and you want the process to keep going.

Related

Image File Execution Options, legit or malicious use?

I am trying to understand what Image File Executions Options can do and what it's normally used for. I know it is used for debugging purposes but how do I know its used for malicious purposes? Is this registry key used with every executable? If not, is it suspicious?
For example what does \Image File Execution Options\DevOverrideEnable do?
PS: I have looked a these sources already:
https://gooroo.io/GoorooTHINK/Article/17352/Image-File-Execution-Options-Good-Evil-Fun/28441
https://blog.malwarebytes.com/101/2015/12/an-introduction-to-image-file-execution-options/
Normally debuggers can run a process or attach to existing process to debug it and it's enough for normal projects but in some projects like services debugger can't start the service because of Windows service architecture. in this scenario we should attach debugger to running service process but if we want to debug some codes in startup of service we can't do it by this method because debugger attached to process after the startup codes executed.
In scenarios like this we can use Image File Executions Options (IFEO) by configuring to launch debugger when process start, when specified process start Windows launch process inside specified debugger and let developer to debug the process from beginning of born !
The problem of this option is Microsoft Windows do not (and cannot ;) check the specified debugger is really a debugge.
On the other hand malware authors use this option as a trick to start they malicious process when some process run, for example they can run a injector every time "explorer.exe" process run and inject some malicious code to it.
Finally IFEO is not malicious option by it self but maybe! and there is not simple way to know it's malicious or not. if you want to be sure it's malicious or not the debugger should be analyze and check manually for malicious activity.
If you are not computer skilled person the easiest way for you is to find the IFEO registry keys and find debugger's path from theme and upload debugger file to online scanner website like VirusTotal and hope they tell the truth :))

Using VMMap in a batch script

I am doing some analysis work on some software we are running where I work. The software seems to have memory issues some where along the line which are proving difficult to track down. We have decided to use Sysinternals VMMap to track the memory being used by the software.
We have VMMap exporting the usage every 20 seconds using Windows scheduler to launch a batch script which pulls back the target process PID and launches VMMap with it. The process runs for a while, output appearing the out directory but after a while it stops. Windows scheduler reports the job ran fine and will start another instance when the trigger is meant, once again with no output.
After a bit of investigation it looks like VMMap is failing to open the process and is trying to report an error through its GUI. Since we are running in batch, we cannot see this error to dismiss it. This is causing numerous process' to be spawned but not actually doing anything.
Has anyone come across this issue when using VMMap, or know of anything that may help? I am thinking there may be some flag I can pass which suppresses messages or maybe some way I can handle it in the batch but Google hasn't helped nor has the Sysinternals forum. Any help would be really appreciated.
VMMap is a GUI tool, so trying to capture its output in an automated way will be difficult. Instead, try using another SysInternals tool, Handle, that captures a lot of the same information, but exports/reports on it in command line, where it can be captured much easier. Alternatively, don't run the output in an auto-repeating way when using VMMap, but instead have your script somehow detect the error or missing expected results/data and stop so the GUI output can be examined.
All Sysinternals tools do pop up a consent dialog for the first time they are started on a new machine to accept their license. I think you did deploy the tool to a production machine and it was trying to show the consent dialog but nobody did press ok.
They do basically create a registry key on the machine which you can fake if you need a fully automated deployement or you can start in once on the target machine for the user in question.

Log DLLs loaded by a process

I'd like to add logging to our unit tests that records the DLLs they use, and where they're loaded from.
I can get the information I need from Sysinternals ListDLLs, but I'd need to run that while the test process is running, and I'd end up with race conditions: for instance, ListDLLs could run too early, and miss a DLL that's loaded half-way through the test run; or ListDLLs could run too late, after the test process exits.
Similarly, I can get the information I need from the Visual Studio debugger's Output and Modules windows, but I'd like to automate this on our build server.
Is there any command line tool that can run an arbitrary EXE, track the DLLs it uses, and log the information to a file?
You may write your own tool, which will use "debugging" features. This tool must
Start new process suspended
Attach to created process as debugger
Process debugging events, as I remember, you need LOAD_DLL_DEBUG_EVENT
http://msdn.microsoft.com/en-us/library/windows/desktop/ms679302(v=vs.85).aspx
The good news: it's not too hard to write it yourself using Detours. Hook the LoadLibraryA/W functions and log DLL names to a file (using GetModuleFileName against the value that the real LoadLibrary returns). Also hook CreateProcess, so that you can log DLLs loaded by child processes.
The bad news: I'd like to be able to post the source code that I used, but it's an internal tool that I won't be able to share.
Edit: I'm not convinced that this tool's Detours hooks are completely reliable, as during my testing, it's missed a few DLLs. Here's an alternative tool using the debugger API: https://github.com/timrobinson/logdlls
Note that SysInternals (now MSFT: http://technet.microsoft.com/en-US/sysinternals) has a great tools for tracking all sorts of events happening when loading your application: Process Monitor. You will have to filter out anything that is not related to the application you are examining. Also, you may want to set Operation="Load Image" filter.
Tried Tim Robinson's tool, but it seems it will only track Windows related dll's so not useful in my case.

Deliberately crashing an external process under Windows

I would like to synthesise a native code fault. This is so that we can see where in particular some debugging output gets put when that occurrs.
Pskill (from Sys-Internals) causes a graceful exit. DotCrash.exe doesn't seem to be available anymore from Microsoft directly.
Is there any way to externally cause a crash in a process?
I've done this before using windbg by:
Starting the process
Attaching to the process with windbg
Setting a breakpoint on one of my app's functions
Running the app until I hit the breakpoint
In windbg setting a local variable to something that will cause an Access Violation (e.g. set a pointer to 0xFFFFFFFF or muck with the register values)
hit f5 and the app should hopefully crash
If what you want is the equivalent of a coredump, drwtsn32 -p ProcessId generates a dump of the current state of a running process. If you have the appropriate debug symbols you can get valuable information.
HTH.
As Nick mentions, this can easily be done via Debugging Tools for Windows - I'd go one step further though, and use cdb (the command-line WinDbg) to script the whole interaction.
If you need dumps at any desired time, you can use Microsoft's free debug diagnostic tool which has a nice UI to do that or on command line drwtsn32 -p processid as recommended by jrbjazz.
You could try using CreateRemoteThread. Using it correctly isn't easy, but making the other process crash should be pretty easy ;-)
Could you install some kind of hook function, or use something like the detours library?

Stop VB application from running in background

I have a console application (written in VB6 ) which is behaving strangely on my machine. I kick it off from the command line and what should be a two minute job drops straight back to the prompt - if I run this on another machine the executable will sit and wait until the job finishes before returning control back to the prompt. If I check process explorer I can see that the executable is running as a background process and other than this strange background-ness is running as expected.
Any thoughts on why this could be happening? (Running on 32-bit Windows XP Pro SP3.)
It's totally unclear whether this is an application you wrote and have the source code for. If that's the case, you need to get in and start debugging. At the least, use OutputDebugString to send information about what's going on to any number of potential viewers. Taking that a step further, consider rewiring the app using the Console module I wrote, along with vbAdvance to recompile. This combination will allow you the full power of the VB6 IDE to debug within. No more guessing about what's going on.
Then again, if it's not your app, I'm not sure what VB6 has to do with it and wish you the best of luck trying to figure out what's up.
It sounds to me as though the app isn't being recognised as a console app on one of your machines. Console apps weren't officially supported in VB6, although there are some well-known hacks for creating them (particularly the free add-in vbAdvance). Possibly your console app is a bit unreliable? If Windows thinks your app is a GUI rather than a console app, it won't wait for it to finish.
As a pragmatic workaround: try launching with start /wait rather than just using the exename. That forces the command prompt to wait for the program to finish, whether it's a GUI app or a console app.
Sounds like an error is occurring that is being 'swallowed' by the application. Do you have the source code?
Errors in VB6 apps are often due to some COM component not installed and/or registered.
Download SysInternals Process Monitor and this will show up accesses to ProgIDs that fail (uninstalled/unregistered COM components).
Check out: Process Monitor - Hands-On Labs and Examples.
Have you checked permissions? Is the application accessing any network based resources?

Resources