VB6 quitting while debugging with active Windows message hook - winapi

One of our largest old VB6 apps has some code in it to allow other apps (including some dotNET ones) to pass an ID to it via a Windows message - this ID is then used by the VB6 app to load an entry in a regular Windows form. The message hook is added after the user is logged in and authenticated, and removed once they logout.
Public Sub HookClaimFinderCall()
lpPrevWndProc = SetWindowLong(gHW, GWL_WNDPROC, AddressOf WindowProc)
End Sub
Public Sub UnhookClaimFinderCall()
Dim temp As Long
If gHW <> 0 Then temp = SetWindowLong(gHW, GWL_WNDPROC, lpPrevWndProc)
End Sub
Private Function WindowProc(ByVal hw As Long, ByVal uMsg As Long, ByVal wParam As Long, ByVal lParam As Long) As Long
If uMsg = WM_FINDCLAIM Then
MasterFindClaim lParam
End If
WindowProc = CallWindowProc(lpPrevWndProc, hw, uMsg, wParam, lParam)
End Function
However, there are two issues with this. The first relates to Visual Studio 6. If the code is being debugged, and an error occurs to bring up the "Continue End Debug Help" dialog box, pressing End quits Visual Studio instantly (losing any unsaved changes). This does not happen if the message hook has not been activated yet. What causes this, and is there anything I can do to stop it short of commenting out the code that loads the hook?
Secondly, if the users quit the app without logging out properly (by whatever means), what happens to the message hook?
I hope I've got all the terms right in the above...

You are going to get crashing (or disappearing VB6) if you initiate a global hook and attempt to Debug in VB6. VB6 is kind of like a simulator, it doesn't exactly duplicate the runtime of a VB6 application and hooking is one of the areas it fails miserably at (though it can't really be blamed if you understand what is going on). For all of the global hooks we use in our applications we check to see if VB6 is running in IDE mode (there are several ways to do this) and if it is, don't run the global hook. If you absolutely have to run the global hook, do not stop the application in the debugger - use debug.print or some other means, but don't stop the application otherwise you will have milliseconds to seconds before it "goes away".
Although you should unhook before exiting the application, when the message pump hits the hook and the handle of the application where the hook takes place no longer exists, it is a fairly cheep operation to ignore that hook. Now if you ran the application and exited thousands of times, it would probably build up, but that I think is the least of your concerns.

Kris's answer is correct. A couple of additional points.
Strictly this is subclassing not message hooking.
It's also good practice in your WindProc to detect WM_NCDESTROY and unhook when that message is received. The message means the window is about to be destroyed - it should be received if the user quits the app, no matter how they do it.
There are some API calls in Windows 2000 and later that make it easier to manage subclassing. As always, Karl Peterson has an excellent article with excellent VB6 code here.

Related

SetWindowsHookEx(WH_SHELL, ...): What is the meaning of event HSHELL_WINDOWREPLACED?

If I register a hook via SetWindowsHookEx(WH_SHELL, ShellProc, ...), what is the meaning of event HSHELL_WINDOWREPLACED? (My Google-fu fails me. I have searched high and low!)
Win32 Docs:
SetWindowsHookEx(): https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-setwindowshookexw
ShellProc (callback): https://learn.microsoft.com/en-us/windows/win32/winmsg/shellproc
The offical docs read: A top-level window is being replaced. Weirdly, they also say: Windows 2000: Not supported. Does that mean only supported before or after Win2K?
I created a test driver to watch a Microsoft Windows session, but I was never able to trigger this mysterious event.
I also found a similar event here:
RegisterShellHookWindow: https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-registershellhookwindow
... that says:
HSHELL_WINDOWREPLACING: A handle to the window replacing the top-level window.
HSHELL_WINDOWREPLACED: A handle to the window being replaced.
Related:
How can I be notified when a new window is created on Win32?
Why HSHELL_WINDOWDESTROYED, HSHELL_WINDOWCREATED?
In this instance, the term "replace" refers to the occasions when a window stops responding to messages ("hangs") and, after a certain period, Windows hides it and replaces it on-screen with a faded-out copy (called a "ghost window").
Windows does this so that, even when the app is not processing messages, the user can interact with the ghost window to move it around and try to close it.
The wParam value is the handle of the hung window (the one being replaced) and the lParam value is the handle of the ghost window (its replacement).
If the window starts responding again, the notification is sent again, with the window handles swapped around.

Why does WM_CLOSE/DESTROY only partially close my Notepad window?

My scenario is that a program calls my DLL and I use Process.Start(notepad) to start a Notepad window on Windows 10 x64. I don't save the process ID of the started process. Then sometime later, a program calls my DLL again, and I find the Notepad window handle (by matching title strings).
The problem is that when I use the handle to send the window a WM_CLOSE or DESTROY message, the job isn't completed. The window does disappear from my screen. It does disappear from the taskbar. But when I look at windows with Alt-TAB, there it is. It's not really gone; it's just hidden from the taskbar. I'm using this Win32 API call to try to close the window.
[DllImport ("user32.dll", SetLastError = true)]
public static extern bool CloseWindow (IntPtr hWnd);
(1) What am I doing wrong?
(2) Are my expectations out of line? Is it even possible for me (as a DLL running under one process) to order Notepad in another process to close?
I suppose, having the Notepad handle in hand, that I could bring it to the foreground and send Alt+F4 to its keyboard buffer somehow to fake it into thinking that I was typing characters to it. But, that seems like the long way around.
(3) How can I programmatically tell apps to close their windows without bringing them to the foreground and sending them keystrokes, or without sending them mouse clicks on the X close button?
I've looked at other posts on the forum, but they mostly talk about terminating the process with the process APIs - a brute force kill method that isn't really what I want to do. (Besides, I want to close the single window that I'm interested in, not a whole process that might be running a dozen different windows like MS Word...)
I'm using [CloseWindow] to try to close the window.
CloseWindow doesn't close a window -- it minimizes it.
Instead, send the window a WM_CLOSE message:
SendMessage(h, WM_CLOSE, 0, 0);
Try doing this:
SendMessage (hWnd, WM_SYSCOMMAND, SC_CLOSE, 0); // or PostMessage perhaps better
This is what Windows sends when you click on the Close box and is much more likely to work in a wider range of apps since the app will then believe that it has been closed by the user and should act accordingly.

SetWindowsHookEx with WH_JOURNALRECORD hook in Qt4

I am debugging a closed source legacy application that uses Qt 4.8.6
The application developed a problem after an automatic update of Windows 10
When the problem is triggered application (and also whole Windows desktop) stops receiving keyboard and mouse events, but applications continue to run, mouse cursor also moves.
Everything goes back to normal if Ctrl+Esc is pressed to open the start menu, until a specific action is done in the application which triggers the problem again.
I traced the problem to Qt's QWidget::grabMouse() which is called on a custom widget in the application.
Tracing the execution of QWidget::grabMouse() shows that the problem happens when it executes:
journalRec = SetWindowsHookEx(WH_JOURNALRECORD, (HOOKPROC)qJournalRecordProc, GetModuleHandle(0), 0);
Its qJournalRecordProc looks like this:
// The procedure does nothing, but is required for mousegrabbing to work
LRESULT QT_WIN_CALLBACK qJournalRecordProc(int nCode, WPARAM wParam, LPARAM lParam)
{
return CallNextHookEx(journalRec, nCode, wParam, lParam);
}
full source of QWidget.cpp where all of this is defined can be found here
Googling reveals that SetWindowsHookEx has special requirements since Vista in order to deter malware from using it. The app seems to fulfill the requirements that I found by googling (signed by trusted certificate although it is using SHA1, installed in "Program Files",...)
And now for the actual questions:
Why Qt needs WH_JOURNALRECORD to perform mousegrabbing? I thought that by using WH_JOURNALRECORD the hook procedure gets mouse/keyboard events and this does not affect the individual widgets. I patched QtGui4.dll so it does not call SetWindowsHookEx (and related Unhook). This fixes the app and does not have any noticeable side effects.
Why would using the WH_JOURNALRECORD hook in this way stop keyboard/mouse events being delivered? I also set a breakpoint on the hook procedure and it seems that it never gets called. I also rigged the hook procedure so it would crash the app (in case this strange hook behavior messes up the debugger) and the app did not crash which in my opinion confirms that the hook procedure is never called.
All this hooking stuff looks like a real ugly hack...
And apparently Qt developers recognized this since Qt5's implementation does not use SetWindowsHookEx anymore...
Edit (to clarify some comments):
I already modified QtGui4 library's grabMouse to not call SetWindowsHookEx and not to call the corresponding UnhookWindowsHookEx.
Since the closed source app apparently uses Qt libraries under the commercial license, they made some closed source modifications to Qt, which forced me to actually patch their QtGui4.dll (changing the part which calls the SetWindowsHookEx into NOPs.
This resolves the problem with the app and produces no noticeable side effects. But I would like to know why Qt developers deemed necessary to put the hook (which does nothing) there in the first place. And what makes it misbehave.
You say you are patching QtGui4.dll but your SetWindowsHookEx line contains GetModuleHandle(0) indicating that qJournalRecordProc is in your .EXE!
Global hooks should be implemented in a .DLL and the correct HINSTANCE should be passed to SetWindowsHookEx.

Detect when a specific process creates a window (on Windows)

I have an application which I automate and run in invisible mode. However I want to detect if that application creates and shows any windows so I can interact (or just hide) them.
I'm looking for a way to get notified by the OS that a window was created by the shell.
(I'm using C#)
This is possible by creating a system wide Windows hook.
Call SetWindowsHookEx with WH_SHELL as the hook id.
In the ShellProc callback function the nCode parameter will have the HSHELL_WINDOWCREATED value whenever a window is created.
Then I can use GetWindowThreadProcessId to check if the window handle provided in the wParam belongs to the process I am interested in.
This must however be implemented in a native dll since .NET assemblies can only implement system wide hooks for WH_MOUSE_LL and WH_KEYBOARD_LL.

Receive screensaver notification

I want to receive a notification in my C++ application when a screensaver is about to start. I tried listening to WM_SYSCOMMAND messages with wParam == SC_SCREENSAVE which some people think should do the trick.
That didn't work. Spy++ even showed that my window didn't receive any WM_SYSCOMMAND message. Interesting thing is when I turned off the monitor I did receive the message with wParam == SC_MONITORPOWER. Am I understanding it wrong? Or did I just miss something?
Edit: For testing I used the default windows screensaver (the one with windows logo).
It appears that I will receive the SC_SCREENSAVE message only when my window has focus. The way around this is to set global hook. That would require me to put the callback function in a separate DLL and there is also this scary message about hooks slowing down the system so I decided to drop the idea of responding to screensaver start.
This is a relatively complex task (although it would be nice if it were easy).
Some of these tests you'll find online only work if your window is in focus. If it's running in the background it may not receive such messages.
Other tests rely on a screensaver program running (check the currently set screensaver, and then watch the process list to see if it's active) but don't work if you go into powersave mode, or if your screensaver is a black screen (ie, no program, just monitor off).
I don't believe there's an ideal way to do this. You might want to go back to the beginning and think more carefully about why you need to detect this state, and what you are trying to accomplish. You might need a different solution.
Probably my answer comes too late.
The MSDN handles screensavers under "Legacy".
On a notebook they waste battery and on a PC they are also useless.
It is better to turn the monitor off than letting it show a screensaver.
As you don't explain exactly what you want to do I don't know if you really need the notification BEFORE the saver starts or if it is enough to get notified when it already has just started.
In the latter case it is easy.
Write a thread that periodically checks:
BOOL b_SaverRunning;
SystemParametersInfo(SPI_GETSCREENSAVERRUNNING, 0, &b_SaverRunning, 0);

Resources