Attaching the debugger to a process when it starts up - visual-studio

I am trying to trace a bug in our handling of com objects. Our program is launched from our activeX control, and there is some com issue when the app is closed again. I would like to attach the visual studio (2008) debugger to our app when it is first launched from the OCX. How can I do that?
Some details:
a) OCX and app written primarily in c/c++ with just a little c++/cli (that play no role in the launching of the app)
b) If I attach to the program while it is running I get an assertion on exit in olelock.cpp
void AFXAPI AfxOleUnlockApp()
{
AFX_MODULE_STATE* pModuleState = AfxGetModuleState();
ASSERT(pModuleState->m_nObjectCount != 0); <--- Triggers this
if (InterlockedDecrement(&pModuleState->m_nObjectCount) == 0)
{
// allow application to shut down when all the objects have
// been released
::AfxOleOnReleaseAllObjects();
}
}
So I would like to attach the debugger to our program when it is first launched from the OCX so I can attempt to track the lifetime of the COM objects. Or perhaps there is another way to trace these objects?

gflags is what you would look into for complex scenarios in general. Specifically this page for debugging an app unconditionally when it is launched.
However, I think this is overkill for what you want. I usually just dump some code in my program that will wait for a debugger to be attached if it was built in DEBUG mode and the F1 key is being pressed when the code starts up. Code like this shouldn't get left in, but it's pretty handy.
#ifdef _DEBUG
if ( ::GetAsyncKeyState( VK_F1 ) )
{
while ( !::IsDebuggerPresent() )
::Sleep( 100 );
::DebugBreak();
}
#endif

Related

Debugging a (marshalled) RCW proxy blocks the message loop on the COM server - why?

I have a simple process that accesses the Excel application object. It subscribes to an event and then attaches a debugger to itself.
The entire source code of the app is as follows:
void Main()
{
var excel = (Application)Marshal.GetActiveObject("Excel.Application");
// if any event has a subscription, attaching the debugger
// freezes the message loop in Excel while the debugger is paused
excel.WorkbookNewChart += (a,b) => {};
Debugger.Launch();
}
When the debugger (e.g. Visual Studio) starts, it freezes (all the threads in) my console app.
What I find puzzling is, this also blocks the message loop in Excel, making Excel unresponsive. However, the message loop in Excel is only blocked if I subscribe to some event, e.g. WorkbookNewChart. It doesn't matter which event and it doesn't matter if the event is fired or not.
My questions are:
Why might attaching the debugger to this console app freeze Excel's message loop and is there something I can do prevent it?
I'm only seeing the behavior of Excel as a black box - are there any tools I could use to figure out what might be blocking the message pump in the Excel process?
[Update: running the same three lines of code in a WPF window's Loaded event handler has the same outcome.]
Background - why I'm interested in this:
I'm making an IDE inside Excel which can be used to build small apps. These apps run in their own processes but access Excel via Marshal.GetActiveObject. My IDE is an Excel add-in running inside the Excel process. I'd like my IDE to be able to debug the runner processes that I spawn from the IDE. Things seem to work ok when starting the apps, but I get deadlocked in situations like the one above, where the debugger freezes the runner process which then blocks the excel process, which makes me unable to use the debugger.
I can, of course, run my debugger/IDE in another separate process, but from a user experience perspective I'd love to keep it inside the Excel task pane if possible.

JIT Debugging hiding bug in C# app

I have an application that works fine under visual studio.
However, when I run it stand-alone, a certain operation which throws an exception isn't handled. (The exception is expected, but it's correctly handled under VS.)
I added a line to my machine.config to enable JIT debugging, to try and locate the problem, but I then could not recreate the bug (the exception was handled correctly). Deleting the line causes the bug to come back.
How should I start looking for the cause?
The relevant parts (I believe) of my code (edited to remove extraneous detail) are at http://pastebin.com/i2zLCTn5.
Some suggestions that may affect how your code is behaving in the different environment of Visual Studio:
Stop Visual Studio turning off the JIT Optimizer when you run your program under Visual Studio.There's an option called "Suppress JIT optimization on module load"...you want to try changing that to be NOT ticked.
For more details on what the option does see here:
http://msdn.microsoft.com/en-us/library/ms241594.aspx
Turn off the "hosting process" ...http://msdn.microsoft.com/en-us/library/ms185330(v=vs.80).aspx
Run the Program, then "Attach to Process" (it then won't be in a hosting process, and the modules will have been loaded and JIT optimized).
In the Exceptions dialog tick the "Thrown" check box of the Exception that is being eated/handled....so that you can track down who is handling it and why. You will need to install the NET Framework Sourcecode if the Exception is being handled there so that you can see the details.
You could have a try at running it under WinDBG instead (it may behave differently as you then won't be running under a hosting process i.e. .vshost)....make sure you have the extension DLL PSSCOR4 (if using .NET 4) so that you can make sense of things.
So, I discovered - or rather, a friend explained - the cause of this problem:
You cannot catch exceptions across threads when JIT debugging is disabled, even if execution appears to flow correctly.
The exception was being thrown in a Form.Closing event handler:
form.Closing += new delegate
{
switch(form.DialogResult)
{
case DialogResult.OK:
// do stuff;
break;
case DialogResult.Cancel:
throw new AbortOperationException();
}
}
// ...
try
{
mainForm.Invoke(new Function<Form, DialogResult>(form.ShowDialog), mainWindow);
}
catch (AbortOperationException)
{
// handle abort
}
The solution was to refactor slightly to eliminate the throw:
form.Closing += new delegate
{
if (form.DialogResult == DialogResult.OK)
{
// do stuff
}
}
// ...
var result = (DialogResult)mainForm.Invoke(new Function<Form, DialogResult>
if (result == DialogResult.Cancel)
{
// handle abort
}

trying to display UI in C++ DLL from VB6 causes assert at AfxGetResourceHandle

I have a legacy vb6 code that calls into c++ to display MFC dialog.
Calling AfxGetResourceHandle to load the dll's resource onto the memory causes debug assert.
Has anyone done this?
HINSTANCE priorRes = AfxGetResourceHandle();
if (m_MDCommonDll == NULL)
{
m_MDCommonDll = LoadLibrary("MyDll.dll");
}
AfxSetResourceHandle(m_MDCommonDll);
If this is not possible to do, is there a way to detect if the caller is from vb6 so that I can skip showing the UI?
Thanks
AfxGetResourceHandle( ) can throw a debug assert if you mix release and debug build DLLs. Make sure your .exe and any .dlls it loads are all built as either release or debug.

Debugging application when launched via ShellExecute

I am trying to launch an application via the ShellExecute() API call. This application contains only a main function which does some processing and exits.
Now I have put DebugBreak() in starting of main. When ShellExecute() is called the application is launched successfully but it does not ask for breaking.
How can I debug my application when launched from other application using ShellExecute()?
I am using VC++ .
If DebugBreak() isn't workign for you, try _CrtDbgBreak(). Note that _CrtDbgBreak only works in a debug build.
_CrtDebugBreak definitely works for me to make a launched process break on startup, although I'm pretty sure DebugBreak does also.
Note that both functions will make it look like the process has crashed, since they raise an exception. That is normal and gives you the opportunity to attach a debugger via the crash dialog. (The crash dialog also lets you terminate the process; don't use that, obviously.)
Also note that if you have a catch-all SEH exception handler around your main then the exception raise by DebugBreak & friends will be swallowed up and the app will simply exit without showing the crash dialog or letting you attach to it.
You can't do this with VC++; with WinDbg this is just .childdbg 1 to debug all child processes. With VC++, you can use Image File Execution Options in a pinch - check out http://codereflect.com/2009/09/20/how-to-debug-child-process-using-windbgvisual-studio/ for more info. Really though, if you've got the time to learn WinDbg, it's much better at this.
you can try this, it's ok in xp system.
app.exe is your application name,
-s1...-s3 is command line arguments.
HINSTANCE hRet = ShellExecute(NULL, L"open", L"vsjitdebugger.exe", L" app.exe -s1 a1 -s2 a2 a3 -s3", szExePath, SW_SHOW);
There is now a Microsoft Child Process Debugging Power Tool.
The method I use for things like this is to embed some interactive code, which you can either delete afterwards, comment out or conditionally enable. In a few cases we have this code enabled by querying an environment variable which is set by the tool that launches the main application. This allows me to click a check box, hit launch and have the breakpoint dialog in seconds.
if (MessageBox(NULL,
_T("Attach the debugger now, then choose Yes to hit a breakpoint"),
_T("Attach Debugger"),
MB_YESNO) == IDYES)
__debugbreak();
This gives you the ability to attach the debugger when the dialog box appears and the option to hit a breakpoint or not. My earlier versions didn't give me the option and after a while I realised some of the time I wanted the breakpoint, and some of the time I didn't.

How to detect an unmanaged app has aborted

I have a C# app which invokes an unmanaged C++ application via Process.Start()
On some machines, if this C++ app aborts, I'm left with the Just-In-Time error dialog showing
Is there a way from C# to detect that the C++ app has errored and just restart it
(I don't have the source to and therefore can't modify the C++ app)
The JIT debugger dialog is showing because the C++ app crashed in some way. Whilst the dialog is open, the process is still present (but suspended) which means that the Process class wont consider it to have "exited" until the JIT dialogue is dismissed or an attached debugger ends the process.
You could either disable JIT debugging on the machines experiencing the problem (I think that disabling the Machine Debug Manager service may disable JIT debugging), allowing the app to crash and die (and, therefore, report as "exited"), or (and I've no idea how you would go about doing this) see if you can make your host C# app attach itself as a debugger for the C++ app and receive notification of the crash from the C++ app, terminate it, and then act upon the information it has ended.
If the C++ is a console based you can check two things,
The exitcode, commonly 0 means closed normally otherwise is problem (but I Think this depends on the convention used to develop this applications)
Check the standard Error if it is empty, so everything is ok
using(Process p = new Process())
{
p.StartInfo.UseShellExecute = false;
p.StartInfo.RedirectStandardError = true;
p.StartInfo.FileName = "any.exe";
p.Start();
string error = p.StandardError.ReadToEnd();
p.WaitForExit()
if(error.Length == 0 && p.ExitCode == 0)
{
}
}

Resources