Debugging application when launched via ShellExecute - winapi

I am trying to launch an application via the ShellExecute() API call. This application contains only a main function which does some processing and exits.
Now I have put DebugBreak() in starting of main. When ShellExecute() is called the application is launched successfully but it does not ask for breaking.
How can I debug my application when launched from other application using ShellExecute()?
I am using VC++ .

If DebugBreak() isn't workign for you, try _CrtDbgBreak(). Note that _CrtDbgBreak only works in a debug build.
_CrtDebugBreak definitely works for me to make a launched process break on startup, although I'm pretty sure DebugBreak does also.
Note that both functions will make it look like the process has crashed, since they raise an exception. That is normal and gives you the opportunity to attach a debugger via the crash dialog. (The crash dialog also lets you terminate the process; don't use that, obviously.)
Also note that if you have a catch-all SEH exception handler around your main then the exception raise by DebugBreak & friends will be swallowed up and the app will simply exit without showing the crash dialog or letting you attach to it.

You can't do this with VC++; with WinDbg this is just .childdbg 1 to debug all child processes. With VC++, you can use Image File Execution Options in a pinch - check out http://codereflect.com/2009/09/20/how-to-debug-child-process-using-windbgvisual-studio/ for more info. Really though, if you've got the time to learn WinDbg, it's much better at this.

you can try this, it's ok in xp system.
app.exe is your application name,
-s1...-s3 is command line arguments.
HINSTANCE hRet = ShellExecute(NULL, L"open", L"vsjitdebugger.exe", L" app.exe -s1 a1 -s2 a2 a3 -s3", szExePath, SW_SHOW);

There is now a Microsoft Child Process Debugging Power Tool.

The method I use for things like this is to embed some interactive code, which you can either delete afterwards, comment out or conditionally enable. In a few cases we have this code enabled by querying an environment variable which is set by the tool that launches the main application. This allows me to click a check box, hit launch and have the breakpoint dialog in seconds.
if (MessageBox(NULL,
_T("Attach the debugger now, then choose Yes to hit a breakpoint"),
_T("Attach Debugger"),
MB_YESNO) == IDYES)
__debugbreak();
This gives you the ability to attach the debugger when the dialog box appears and the option to hit a breakpoint or not. My earlier versions didn't give me the option and after a while I realised some of the time I wanted the breakpoint, and some of the time I didn't.

Related

Invoke debugger programmatically?

Is there a way to invoke the script debugger programmatically?
I want to create a dialog box that offers an option to start the debugger (as if it had reached a breakpoint in one of the callers), similar to the one that QTP offers when it encounters a playback error.
This is for QTP, but i estimate that if one can do this for VBS, it'll work there too.
Updated update
The stop statement does what I want -- except for this (in conjunction with QTP/UFT):
it stops where the stop statement is (and not in the caller's code) (I think I could work around that by putting the function with the stop statement into a file that I load in a way that it won't be jumped into by the debugger (LoadLibrary? ExecuteFile?));
it requires me to have triggered one "real" breakpoint during the run session previously, and continued execution with "Run"/F5.
Does Err.Raise work for what you're trying to achieve?

Execute Till User Code doesn't work

I'm trying to use OllyDbg's "Execute Till User Code" feature (which is essential for me) but it never works.
I first tried it on a program which called MessageBoxA. When it called it I paused the program in the debugger and issued OllyDbg to execute till user code, but the program was still paused and completely frozen. I couldn't even click the MessageBox's OK button, or even make it continue from OllyDbg. When I tried too hard to make it continue it just crashed.
So I decided to write an application in NASM (to have complete control) and tried it there. The MessageBoxA popped, I paused, executed till user code, I could press the OK button this time, but OllyDbg didn't pause the program once I did. The program just executed as if nothing happened.
Why doesn't it work and what can I do to solve it?
OllyDbg 1.10 (No Plugins)
Windows 7 Ultimate SP1 64-bit
Update:
I've tried to use OllyDbg 2.0 but the Execute till user code button is completely disabled for some reason.
I ran into the same issue while following Lena's tutorial #4 and I think I've figured it out. When I press Alt+F9 to "Execute Until User Code," I notice that all but one thread in the Threads window (Alt+T) have the suspend flag set. Evidently, Alt+F9 only resumes one thread. Fortunately, OllyDbg has the ability to manually resume the other threads by right clicking on them and selecting "Resume" (or by using the + key). For me, this was sufficient both to unfreeze the dialog and to get OllyDbg to pause immediately after the call to MessageBoxA.

Visual Studio debugging program locked in memory

When i am debugging my program and an error occur, the debug session ends, but the program remains in memory. Using the activity manager of Windows to close it does not work. I need to close Visual Studio in order to kill the process.
Why is this happening?
When during debugging a program error occurs, the program usually does not "end". Instead, the debugger (VS2010) pauses execution, allowing you to inspect the code resulting in the error. Depending on the language used (e.g. C#) and the way you compiled your program, you may even be able to edit the program on the fly, move the execution cursor back a bit and continue the program from there.
If the Debug toolbar is visible (in my case it shows up automatically whenever I'm debugging), you should see a couple of "playback" buttons, allowing you to start/continue, pause, stop your program etc. If you stop your program, it will be gone from the task manager too.
As I mentioned in a comment on your question, you can also use the Debug menu to accomplish these tasks.

How to start a process and then pause it immediately?

I would like to know if there is any command or tool that can be used to start a process and then pause it immediately.
I need this function so that I can have time to attach a debugger to it. I have tried visual studio's /debugexe feature, but the behavior of the program seemed changed. So I need to find other way to attach it and debug.
Thank you.
You can use CreateProcess() with CREATE_SUSPENDED flag. This will start the process with the main thread suspended. Then you call ResumeThread() after having attached.
In your main routine insert the line:
System.Diagnostics.Debugger.Launch();
Compile in debug mode.
Run your program and it will break and prompt you to attach a debugger. If you have studio open it will ask if you want to use it to debug.
How about adding a Sleep() call as the very first statement.
You can also add a call to
System.Diagnostics.Debugger.Break();
in the code. It will suspend the program and then ask you to choose the debugger you want to use. You can then attach to a running Visual Studio instance. If you're already running from a debugger, it will simply break as if it would have hit a breakpoint.

What is the best way to debug a crashing explorer.exe?

I work for an orginization that has a custom built Access/SQL Application running in house. We have a problem Explorer.exe throwing an error and crashing.
This is a picture of the crash:
What is the best way to start tracking this problem down and finding a solution ?
Make sure WinDBG is installed, set it up as the default debugger then use Analyze and get a crash dump.
The next time you get that dialog click "OK" to attach in WinDbg
Keep in mind that if you attach a debugger to explorer, then break, that you should not try to do anything in the debugger that invokes a call to explorer or you will deadlock the debugger. This can be surprisingly tricky as it means you cannot open a "File Open" dialog, print, browse to another computer and any number of other things.

Resources