Program Freezes on MessageBox() - windows

Here's the problem: The main GUI thread is performing a SendMessage to another GUI thread (yes, there are multiple GUI threads, and unfortunately this cannot change). When that second GUI thread receives the SendMessage, it may decide to display a message box. Some of the time, that MessageBox will 'freeze' the entire application.
More specifically, the message box shows up, but the entire GUI is hung (user input does not work anywhere).
I've verified with a debugger that the second GUI thread is spinning in the DialogBox2() function defined in user32.dll. I can see in the disassembly that a message pump is being executed (I see IsDialogMessage/TranslateMessage/DispatchMessage being called). Using spy++, I do not see any messages being processed for the message dialog box window. I do see messages getting processed on the main GUI window (such as WM_SETCURSOR, though I do not thin they are being processed as I believe SendMessage doesn't execute a message pump).
The second thread is executing code that is part of an MFC extension DLL, if that matters.
I've tried using AfxMessageBox() / CWnd::MessageBox / ::MessageBox(NULL parent window,...). All exhibit the same problem.
Has anyone seen anything similar before?
Thanks,
Andrew

It must be that blocking one of the GUI threads causes the problem.
Try this:
Replace the ::SendMesage with ::PostMessage followed by a ::MsgWaitForMultipleObjects loop. You will need to pass an event handle that signals when the message box is closed.
It will probably solve the problem.
Just be careful which messages you dispatch in you ::MsgWaitForMultipleObjects loop.

Related

Debugging a (marshalled) RCW proxy blocks the message loop on the COM server - why?

I have a simple process that accesses the Excel application object. It subscribes to an event and then attaches a debugger to itself.
The entire source code of the app is as follows:
void Main()
{
var excel = (Application)Marshal.GetActiveObject("Excel.Application");
// if any event has a subscription, attaching the debugger
// freezes the message loop in Excel while the debugger is paused
excel.WorkbookNewChart += (a,b) => {};
Debugger.Launch();
}
When the debugger (e.g. Visual Studio) starts, it freezes (all the threads in) my console app.
What I find puzzling is, this also blocks the message loop in Excel, making Excel unresponsive. However, the message loop in Excel is only blocked if I subscribe to some event, e.g. WorkbookNewChart. It doesn't matter which event and it doesn't matter if the event is fired or not.
My questions are:
Why might attaching the debugger to this console app freeze Excel's message loop and is there something I can do prevent it?
I'm only seeing the behavior of Excel as a black box - are there any tools I could use to figure out what might be blocking the message pump in the Excel process?
[Update: running the same three lines of code in a WPF window's Loaded event handler has the same outcome.]
Background - why I'm interested in this:
I'm making an IDE inside Excel which can be used to build small apps. These apps run in their own processes but access Excel via Marshal.GetActiveObject. My IDE is an Excel add-in running inside the Excel process. I'd like my IDE to be able to debug the runner processes that I spawn from the IDE. Things seem to work ok when starting the apps, but I get deadlocked in situations like the one above, where the debugger freezes the runner process which then blocks the excel process, which makes me unable to use the debugger.
I can, of course, run my debugger/IDE in another separate process, but from a user experience perspective I'd love to keep it inside the Excel task pane if possible.

Hooking message 0x202 in machine code and inspecting the associated window procedure?

I'm trying to figure out what my debugger does when I hit the pause button, so I decided to hook the WM_LBUTTONUP message (by inserting my own code after PeekMessageW and hitting an int3 if the message is equal to 0x202) so that I can see what commands run after that and how it differs from other messages/no message.
Unfortunately there appears to be no significant change in the code path, as it appears to just run "the message loop" either way and I can't find out how to get to where "LRESULT CALLBACK WindowProcedure()" would normally be if I were looking at the code in C/C++. How do I get to that portion of the code in my debugger?

MessageBox prevents exiting from application

Our application loads 3rd party DLLs that sometimes pops MessageBoxes.
We've notices that sometimes we can't just use exit(0) when there's an open MessageBox.
How can we still force an exist in such a case?
Thanks
Seems like your ugly DLL calls MessageBox (or whatever) from within DLL_THREAD_DETACH or DLL_PROCESS_DETACH.
If this happens in the same thread (i.e. the thread the calls exit) you may try to call PostQuitMessage right before the application exit. This should abort any message loop. If this happens in another thread - you may call PostThreadMessage.
There's also an option to intercept calls to Win32 API functions by hooking the appropriate module (Exe/Dll) import table. Invented by J.Richter (if I remember correctly). http://www.player.idv.tw/prog/index.php/APIHook
If you want to use brute force you may call the TerminateProcess() function. I'm not sure this is a good idea, you may want to detect if there's an open message box and send a close to it (for example using a FindWindow())
Take a look at MSDN.

Debugging application when launched via ShellExecute

I am trying to launch an application via the ShellExecute() API call. This application contains only a main function which does some processing and exits.
Now I have put DebugBreak() in starting of main. When ShellExecute() is called the application is launched successfully but it does not ask for breaking.
How can I debug my application when launched from other application using ShellExecute()?
I am using VC++ .
If DebugBreak() isn't workign for you, try _CrtDbgBreak(). Note that _CrtDbgBreak only works in a debug build.
_CrtDebugBreak definitely works for me to make a launched process break on startup, although I'm pretty sure DebugBreak does also.
Note that both functions will make it look like the process has crashed, since they raise an exception. That is normal and gives you the opportunity to attach a debugger via the crash dialog. (The crash dialog also lets you terminate the process; don't use that, obviously.)
Also note that if you have a catch-all SEH exception handler around your main then the exception raise by DebugBreak & friends will be swallowed up and the app will simply exit without showing the crash dialog or letting you attach to it.
You can't do this with VC++; with WinDbg this is just .childdbg 1 to debug all child processes. With VC++, you can use Image File Execution Options in a pinch - check out http://codereflect.com/2009/09/20/how-to-debug-child-process-using-windbgvisual-studio/ for more info. Really though, if you've got the time to learn WinDbg, it's much better at this.
you can try this, it's ok in xp system.
app.exe is your application name,
-s1...-s3 is command line arguments.
HINSTANCE hRet = ShellExecute(NULL, L"open", L"vsjitdebugger.exe", L" app.exe -s1 a1 -s2 a2 a3 -s3", szExePath, SW_SHOW);
There is now a Microsoft Child Process Debugging Power Tool.
The method I use for things like this is to embed some interactive code, which you can either delete afterwards, comment out or conditionally enable. In a few cases we have this code enabled by querying an environment variable which is set by the tool that launches the main application. This allows me to click a check box, hit launch and have the breakpoint dialog in seconds.
if (MessageBox(NULL,
_T("Attach the debugger now, then choose Yes to hit a breakpoint"),
_T("Attach Debugger"),
MB_YESNO) == IDYES)
__debugbreak();
This gives you the ability to attach the debugger when the dialog box appears and the option to hit a breakpoint or not. My earlier versions didn't give me the option and after a while I realised some of the time I wanted the breakpoint, and some of the time I didn't.

Is there an equivalent of DoEvents in Visual Studio macros?

I've got a long-running macro, in which I write periodic messages to Output just so that the user know somethings happening. The problem is that the output doesn't get shown until the macro has finished (unless you have the Continue Waiting? dialog box open, strangely enough).
Is there anyway to 'flush' the event queue?
If you reference Windows Forms you can access Application.DoEvents as normal.
I have the main IDE responding to windows updates and closes in real time while the macro is still running with only
DTE.SuppressUI = False
Threading.Thread.Sleep(5000)
As such I suggest using a standard DoEvents & Sleep(55) loop for a 250 milliseconds or so when you want to ensure the UI is updated should do the trick, along with the SuppressUI=False.

Resources