VB6 Service debugging with WINDBG - debugging

First and foremost, I'm an absolute newbie at debugging compiled code (until recently, I didn't even know it was possible!). I've successfully used the VC6 IDE to attach to a process but I can't get WinDbg to work...
The problem is that I have a VB6 service that I need to debug in a production server. I really don't understand much of how the .EXE is compiled because it is generated by an external tool called "NT Service Toolkit". The code I have is compiled into an ActiveX DLL that is used by this .EXE.
I've done everything I found on the internet to make the service debuggable with the VC6 IDE: compile without optimizations and generate symbols. When I use the VC6 IDE to attach to it, it works as expected... I can set breakpoints and everything works fine except for the annoying bug that kills the service when the debugging session stops.
When I try to use WinDbg however, I cannot set breakpoints; it fails with an error:
Unable to insert breakpoint 0 at 00000000`0046f715, Win32 error 0n299
"Only part of a ReadProcessMemory or WriteProcessMemory request was completed."
bp0 at 00000000`0046f715 failed
WaitForEvent failed
I've already downloaded symbols for windows' components from the symbol servers, as instructed in the documentation I could find, and I also included the PDB file for my service in the symbols folder...
One thing I noted is that I can get symbolic information when I check the "Noninvasive" box when selecting the process to attach to because I see things like wow64win!ZwUserGetMessage instead of just the address`offset syntax... but when I do, I also get this warning:
WARNING: Process <PID> is not attached as a debuggee
The process can be examined but debug events will not be received
In this case, I can set breakpoints, but when I try to run the service, it fails with:
0:000> g
^ No runnable debuggees error in 'g'
UPDATE
When I set a breakpoint in "Noninvasive" mode, detach, and reattach with "Noninvasive" un-checked, the breakpoints I set while in "Noninvasive" mode remain, and are hit successfully, but I can't set new breakpoints unless I repeat the whole process (detach, attach in "Noninvasive" mode and reattach). Does this make sense? What could I be doing wrong?

I found the problem. It was a conflict in the module names.
When WinDbg loads the modules for a process, it names them as the file where they live, just WITHOUT extension. Therefore, if, as in my case, two modules that must be loaded share the same name, say MyService.exe and MyService.dll, there will be two modules named MyService; in this case, WinDbg breaks the ambiguity by appending _<start-address> to the name, resulting in, for example:
...
MyService
MyService_11000000
...
(The following is a rationalization of what happened, since like I said, I'm pretty new to "attach to process"-type debugging and I lack knowledge of WinDbg internals...)
My problem was that the PDB file for the DLL (MyService.pdb), which was the module I was interested in debugging, was being matched to the EXE file's module (MyService), so the DLL's symbols were not loaded before trying to "restore" the breakpoints, resulting in the Win32 error 0n299 I was seeing earlier... It appears that the g command also tries to "set" the saved breakpoints, and since the symbols were not yet loaded, it would not let me start the thing after the default breakpoint either...
...
Or something like that... Obviously, if someone has a more accurate explanation, I'd very much like to be enlightened. :)
For the moment, I just changed the service EXE's name, and now everything works fine.

Related

Windows scripting: is it possible to capture missing-DLL errors in a log file?

When you try to run a program on Windows, and the loader can't find all of the required DLLs, the default behavior is to pop up a dialog box that describes the problem, including both the name of the program and the name of (one of the) missing DLLs. The process then hangs until someone clicks OK, and then exits with an error code. Here's an example of this dialog box:
Now suppose you're scripting some automated process that might fail for this reason, e.g. running CI tests after installation, where part of the point is to make sure the installer installs all the DLLs. You don't want your build workers to hang waiting for someone to click on a dialog box that's being displayed on a monitor that the computer physically does not have because it's in a server rack somewhere. You want the test cycle to stop immediately and the details to get written to the log.
Your build driver can disable this dialog box for itself and all its child processes (assuming nobody uses CREATE_DEFAULT_ERROR_MODE) by calling SetErrorMode:
SetErrorMode(SEM_FAILCRITICALERRORS | SEM_NOGPFAULTERRORBOX | SEM_NOOPENFILEERRORBOX);
hProc = CreateProcess(...);
However, this only solves half of the problem. The offending process will terminate with an exit status of 0xC0000135 (STATUS_DLL_NOT_FOUND), but the name of the problem executable and the name of the missing DLL are not reported anywhere that I can find it.
So here's the actual question: From code running in the build driver, how do I get the name of the problem executable and the name of the missing DLL, so that I can write them to my CI build logs? Anything goes, except that the code of the problem executable itself and its DLLs cannot be modified (because this is supposed to be a general solution) and an approach that does not require elevated privileges is strongly preferred.
(This is a follow-up question for Suppress "The program can't start because X.dll is missing" error popup . I've been vaguely meaning to write it for years now.)
Even though it's a little overkill, you could write a program/script that parses the PE format of each executable, walks its import table to enumerate imported modules, and checks to see if each module exists on the system (i.e. is it in \Windows\System32, the current directory, etc).
Then, if the executable in question is missing an import, log the executable and the missing import somewhere (and don't run the executable, of course).
If you want to do this in Python, the pefile module is excellent (https://pypi.org/project/pefile/).
(Update: This won't work for DLLs that are imported via LoadLibrary)

System.Diagnostics.Debugger.Break() has issues breaking in standalone DLL in PowerShell

I build a "hello World" kind of app-dll combination in C# and set the break point
System.Diagnostics.Debugger.Break() in the DLL and it breaks promptly no issues.
I have another exe and DLL combination for which I have the source code for the DLL only. In this new DLL source code I insert System.Diagnostics.Debugger.Break() build the Debug version of the DLL but when it runs in a PowerShell it does NOT break. Why is it not breaking? Further System.Console.WriteLine() works in my sample DLL but not in the real DLL.
I checked the csproj file hoping to find a setting like Break on BreakPoint: NO or Screen Print: NO or something to that affect no luck.
I was loading the DLL inside a PowerShell and then running it and hoping it to freeze up or show some sing of failure. Then after much Googling I realized PS was preventing the break and the display. So the solution is, run WinDbg and open powershell.exe (Open executables). That will open a new PS and in that environment my DLL does break and does display the Console Writes.

Unmanaged C++ regfree COM application: problems when launching from IDE with parent directory in path (VS2010)

I have an application A and a DLL B, both unmanaged C++ built in VS2010. DLL B is a COM server. A uses both implicit C++ linkage to B for some methods and classes but it is also accessing coclasses implemented in B via COM. This all works fine when using registered COM, but when A is using B via registration free COM, a weird problem is occuring.
When I start A from within the IDE (either F5 or Ctrl+F5), then my application crashes when trying to cocreate a coclass implemented in B. I also noticed that when I try to debug this the Visual Studio Debugger sometimes simply ignores my commands to step into methods. The source code and what is actually executing seem to be somehow out of sync.
I then eventually found the cause of the problem: In application A's Debugging settings Command is set to $(TargetPath). $(TargetPath) in turn uses the project's output directory which was set to "$(ProjectDir)..\bin\". Changing the output directory to an absolute path that does not contain the ".." parent directory syntax, solved the problem, even though the directory pointed to was exactly the same.
Can anyone explain what is going on there? I have a vague idea that it has something to do with the fact that my application both loads B.dll at startup via implicit C++ linkage and also lists it as a dependent assembly in its manifest for registration free COM. Could this lead to B dll somehow being loaded twice? And why does this only happen when starting A from within the IDE? When launching A from the command line, using ".." in thee path works fine.

CreateProcess vs Command Line and missing DLL

When I use CreateProcess to start an application (InDesign) that is loading my plug-in, my plugin fails to load because a DLL my plugin uses can't be loaded. The strange thing is that if I double click the application or open it from a command line, my plugin loads fine.
My application that calls CreateProcess is quite large and itself uses several DLLs some of which are also used by my plug-in. I've narrowed it to the DLL that will will not load. I set it to delay load, the plugin now loads but will not load that DLL when I call it.
I've used "Process Explorer" to check out the application's PATH to make sure it's the same as when I open it from the command line. The path to the trouble DLL is in the PATH env, this is the same for other DLLs that load fine.
I launch many different apps with and without my plugins and have never seen something like this before.
Anybody know what might be going on?
Oops...
Well, I was wrong about the env PATH, it was different depending on how the application (InDesign) was launched. It looks like QuickTime changes the PATH for my application, placing it's "Apple Application Support" folder at the top. Inside that folder is a DLL with the same name as the one I'm using, namely libxml2.dll. When I load libxml2.dll in my plugin using LoadLibrary, msvcr80.dll can't be found. Shish.
Sorry for the distraction...
Check the working path when loading the application and/or plugin. Explorer and the command prompt normally set this explicitly but another application calling CreateProcess() may not.

QTAgent32.exe keeps a handle to a DLL open after execution

I have run into a bit of a problem using Visual Studio 2010's unit test framework. Currently the QTAgent32 will maintain a reference to a DLL after the execution of a test has finished.
The DLL in question is a c++/cli wrapper around some native c++ code. The object exposed by the wrapper is actually never assigned to by the managed code. The only reference it makes is a final check during disposal to see if it has been set during the class's life time.
If(_obj != null)
{
_obj.Dispose();
_obj = null;
}
I know this for a fact since if i step through the code and watch the debugger console output I can see the symbols for the DLL are not loaded until hitting the if (which makes sense). All managed objects involved implement IDisposable to make sure that all native objects are taken care to avoid memory leaks.
Based on this question: QTAgent32 Holding File Open I made sure that no filestreams were explicitly opened (including Console) and no files are even used yet the problem remains. I am running out of ideas on what to do.
Can anyone help?
TL;DR: QTAgent32.exe keeps an open reference to a c++/cli wrapper that is never instantiated.
I ran into the same problem while using a Fortran DLL. The problem persisted even if the DLL function was completely empty.
I still don't know what the issue is, but an easy workaround is to specify the killing of the QTAgent32 process as a pre-build event in your project.
taskkill /f /im QTAgent32.exe
exit 0
It probably opens the DLL in metadata only mode in order to search for test attributes. This wouldn't trigger symbol loading, but it could lock the file.

Resources