debugging a dynamically loaded executable - visual-studio-2010

I've been handed an application to support, and I'm trying to figure out how to do it. I do have the source, and can make some changes, but I obviously don't want to completely change the architecture of the application.
The app is in a VS2010 solution composed of 9 different projects. The main one is a Windows Form application, but it spins off others in other threads.
Here's the difficulty. Even though the different projects are parts of the same solution, they are separate executables, not DLLs. When the main program starts one of the other projects, it does so by creating a new process, setting the filename of the executeable, the startup arguments and other assorted parameters into the process.StartInfo object, and then calls process.Start().
How can I set breakpoints and debug subordinate executables? I can't attach to them until they are loaded, but they don't get loaded until process.Start() is called, and by then it's too late. Is there a method call I can insert into the main program to get it to load the executable (so I can set breakpoints in it) before it actually begins execution?
Thanks.

Are you able to recompile the other executables? If so, have you tried putting DebugBreak in suitable places? (or _asm int 3).
You can't load the process (usefully), since by definition it will be run in a different addres space from the one you are debugging before it starts.

One simple solution could be adding a call to DebugActiveProcess function to the "main" function of every process which participates in your application.

Related

Easiest way to remove dependency on certain ActiveX library in MFC project?

I am working on greatly simplifying an old MFC application, and I'd like to use that opportunity to remove a dependency on a certain third-party ActiveX library that has been causing headaches. I first simply tried to remove all references to these controls in the cpp code. It compiles fine, but when installed on a new computer, it crashes if the library .msm file is not included in the setup project.
The problem now is that I don't know where any remaining references to these objects in the project is, and therefore have no easy way of tracking them down and removing them. Simply using the Find-functionality in VS only returns results in cpp files which I've already cleaned up, not any control instances in f.ex. dialog resource files. Is there a way to search for all these objects in the project, or check for them compile-time so that I can use compile errors to see where any library objects remain?
First of all: CoCreateInstance, CoGetClassObject are the normal ways how COM Objects are created. Should be easy to set break points here or to set a hook to monitor what's going on.
Seams that you have a bad error handling in you software. Otherwise you application should show an error message if an object could not be created.
Try to debug.... use remote debugging, even a crash dump will lead you to the code location.
Easiest approach: In your development machine. Try to look into the output window and see if any of the old ActiveX DLLs gets loaded. Trigger down to the location were any of those objects is created.
You can do the same with remote debugging on the test machine.

How do I include an SDK routine in VB6?

I am modifying some old VB6 code and there are a number of calls to a subroutine called SDKPress(Index as integer, PressStatus as Integer, PressX as double, PressY as double, PressDataX as double, PressDataY as double).
(This routine returns the position of the mouse click and Index returns which graph on a form the mouse is in.)
Since there is no code for this in the project, I assume it is from some Windows software development kit. However, there is no trace of it on the development PC. just an SDK directory in the VisualBasic directory containing winsdk_web.exe.
I cannot call that subroutine from another project. Where does the subroutine live? Where can I find documentation on it? How can I get it in a form where I can use it on another project?
To find out where it lives. Put the cursor in one of the routine calls in the IDE, right-click on it, and choose go to definition. See where it takes you.
It may be an ordinary routine in a code file in your project.
It may be a routine in a DLL which is imported into your project via a Declare statement.
It may be a routine in a COM/ActiveX DLL or OCX control or even ActiveX EXE. In which case the IDE will show you the "object browser" window and you should be able to figure out where the routine lives, and go from there. I.e. figure out how to add a reference to it into another VB project, and how to install/register the relevant DLL/OCX/EXE on another PC.
OR, and I'm sure you tried this already, but just in case, search the source code for the routine name. Use Ctr-F or menu item Edit-Find in the VB IDE.
DLL files contains the API functions, vb6 code will load the DLL (Declare Function SDKPress Lib "DLLName" ....) and call the functions, use a tool like DllExpView: https://www.nirsoft.net/utils/dll_export_viewer.html and list DLLs and their functions. Or it also might be an internal subroutine/function.
If the original program which uses the SDK runs, you can gather some information using Process Monitor and Process Explorer. These can help you locate dependencies of many types.
I'd start with the second - when you run your program, ProcExp will show you all the files, including DLLs, that are loaded into the process. This alone might give you some insight.
If that is insufficient then use ProcMon to actively monitor the activities of your program. While ProcExp is basically a snapshot in time, ProcMon will dynamically record many of the actions of your program, including how it searches for and ultimately loads DLLs.
https://learn.microsoft.com/en-us/sysinternals/downloads/procmon
https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer
I have found these tools to be invaluable on many occasions. They help to break down the "black box" around a running process which appears in many programmers minds.

Executable loads same dll as already loaded dll

I'm about to start making major modification to my project and I just want to clarify something as I think my design maybe somewhat complicated.
I have an executable that loads a dll, lets call this dll1, this then loads dll2.
The executable also loads dll2.
What I'm asking is do I have two instances of dll2's global and static member variables, does the second load of dll2 happen or can an execuable only ever load 1 of dll2 even if dll2 was loaded by a different dll?
I know I should only have one copy of dll2's code in memory this is fine. It is the global and static variables I'm interested in.
You can only have one instance of any particular DLL loaded per process.
You can of course load different instances of the same DLL, this is practice is not common, but it is technically possible. Have a try with Process Explorer. See the snapshot below.

QTAgent32.exe keeps a handle to a DLL open after execution

I have run into a bit of a problem using Visual Studio 2010's unit test framework. Currently the QTAgent32 will maintain a reference to a DLL after the execution of a test has finished.
The DLL in question is a c++/cli wrapper around some native c++ code. The object exposed by the wrapper is actually never assigned to by the managed code. The only reference it makes is a final check during disposal to see if it has been set during the class's life time.
If(_obj != null)
{
_obj.Dispose();
_obj = null;
}
I know this for a fact since if i step through the code and watch the debugger console output I can see the symbols for the DLL are not loaded until hitting the if (which makes sense). All managed objects involved implement IDisposable to make sure that all native objects are taken care to avoid memory leaks.
Based on this question: QTAgent32 Holding File Open I made sure that no filestreams were explicitly opened (including Console) and no files are even used yet the problem remains. I am running out of ideas on what to do.
Can anyone help?
TL;DR: QTAgent32.exe keeps an open reference to a c++/cli wrapper that is never instantiated.
I ran into the same problem while using a Fortran DLL. The problem persisted even if the DLL function was completely empty.
I still don't know what the issue is, but an easy workaround is to specify the killing of the QTAgent32 process as a pre-build event in your project.
taskkill /f /im QTAgent32.exe
exit 0
It probably opens the DLL in metadata only mode in order to search for test attributes. This wouldn't trigger symbol loading, but it could lock the file.

What unloads the modules when a vb6 program terminates

The proper way to end a vb6 program is to do something like this :
Dim frm As Form
For Each frm In Forms
Unload frm
Next frm
end
That takes care of forms, what takes care of modules in memory?
There is no need to explicitly unload modules in VB6. They are unloaded automatically when the last form is unloaded. The language doesn't support references to standard modules at all, only to the (global) functions and variables defined therein. Since you can't reference the module, you can't unload them either.
Don't use End, you don't need to (ever). Then all memory gets freed properly.
Even using End, memory should be freed automatically. There was a rumor that some class instances are forgotten and don't get terminated correctly, despite reference counting. Thus, it's been established as best practice to set all object instances to Nothing explicitly (especially, but not limited to instances allocated in modules). I've never seen any confirmation that this is actually true (it might still be, though!).
If by "module" you mean static .BAS modules (they're all modules: Forms, Classes, UserControls, etc.) you don't need to "unload" them because they're static.
Most other module types are dynamically loaded. In the case of Forms typically via the predeclared global reference variable with the same name and type as the Form class (yes, a Form is a kind of class).
Note that if you use global variables then you should check if any of them need to get cleaned up. If so you should have a CleanUp method of your module and call that during the unload event of your main form. Another gotcha is sometimes the ORDER of how you unload things is important. By doing an explicit cleanup you can control that.
In VB6 you should rarely use END. If you have any circular references the program will remain as a process chewing up resource causing various strange bugs when you fire it up again. With COM objects is very easy to in inadvertently setup a chain of object thats is circularly linked.
End exists as a compatibility holdover from previous version of QuickBASIC and Visual BASIC. It did not start causing major problems until VB 4.X introduced the ability to create classes. It started gaining attention in 5.X.
Prior to this one of the only ways to make this happen in VB 3.X is to have two forms set references to each other.
The method of Unloading ALL MODULES on exit had resolved a funny bug for retrieving an exit code with API call ExitProcess()

Resources