SetWindowsHookEx with WH_JOURNALRECORD hook in Qt4 - winapi

I am debugging a closed source legacy application that uses Qt 4.8.6
The application developed a problem after an automatic update of Windows 10
When the problem is triggered application (and also whole Windows desktop) stops receiving keyboard and mouse events, but applications continue to run, mouse cursor also moves.
Everything goes back to normal if Ctrl+Esc is pressed to open the start menu, until a specific action is done in the application which triggers the problem again.
I traced the problem to Qt's QWidget::grabMouse() which is called on a custom widget in the application.
Tracing the execution of QWidget::grabMouse() shows that the problem happens when it executes:
journalRec = SetWindowsHookEx(WH_JOURNALRECORD, (HOOKPROC)qJournalRecordProc, GetModuleHandle(0), 0);
Its qJournalRecordProc looks like this:
// The procedure does nothing, but is required for mousegrabbing to work
LRESULT QT_WIN_CALLBACK qJournalRecordProc(int nCode, WPARAM wParam, LPARAM lParam)
{
return CallNextHookEx(journalRec, nCode, wParam, lParam);
}
full source of QWidget.cpp where all of this is defined can be found here
Googling reveals that SetWindowsHookEx has special requirements since Vista in order to deter malware from using it. The app seems to fulfill the requirements that I found by googling (signed by trusted certificate although it is using SHA1, installed in "Program Files",...)
And now for the actual questions:
Why Qt needs WH_JOURNALRECORD to perform mousegrabbing? I thought that by using WH_JOURNALRECORD the hook procedure gets mouse/keyboard events and this does not affect the individual widgets. I patched QtGui4.dll so it does not call SetWindowsHookEx (and related Unhook). This fixes the app and does not have any noticeable side effects.
Why would using the WH_JOURNALRECORD hook in this way stop keyboard/mouse events being delivered? I also set a breakpoint on the hook procedure and it seems that it never gets called. I also rigged the hook procedure so it would crash the app (in case this strange hook behavior messes up the debugger) and the app did not crash which in my opinion confirms that the hook procedure is never called.
All this hooking stuff looks like a real ugly hack...
And apparently Qt developers recognized this since Qt5's implementation does not use SetWindowsHookEx anymore...
Edit (to clarify some comments):
I already modified QtGui4 library's grabMouse to not call SetWindowsHookEx and not to call the corresponding UnhookWindowsHookEx.
Since the closed source app apparently uses Qt libraries under the commercial license, they made some closed source modifications to Qt, which forced me to actually patch their QtGui4.dll (changing the part which calls the SetWindowsHookEx into NOPs.
This resolves the problem with the app and produces no noticeable side effects. But I would like to know why Qt developers deemed necessary to put the hook (which does nothing) there in the first place. And what makes it misbehave.

You say you are patching QtGui4.dll but your SetWindowsHookEx line contains GetModuleHandle(0) indicating that qJournalRecordProc is in your .EXE!
Global hooks should be implemented in a .DLL and the correct HINSTANCE should be passed to SetWindowsHookEx.

Related

Detect when a specific process creates a window (on Windows)

I have an application which I automate and run in invisible mode. However I want to detect if that application creates and shows any windows so I can interact (or just hide) them.
I'm looking for a way to get notified by the OS that a window was created by the shell.
(I'm using C#)
This is possible by creating a system wide Windows hook.
Call SetWindowsHookEx with WH_SHELL as the hook id.
In the ShellProc callback function the nCode parameter will have the HSHELL_WINDOWCREATED value whenever a window is created.
Then I can use GetWindowThreadProcessId to check if the window handle provided in the wParam belongs to the process I am interested in.
This must however be implemented in a native dll since .NET assemblies can only implement system wide hooks for WH_MOUSE_LL and WH_KEYBOARD_LL.

How to provide a HWND to DirectSound SetCooperativeLevel in a console program?

I'm writing a console program that uses DirectSound API to render some audio data. I stumbled on a curious problem when following the DirectSound Programming Guide (from Microsoft). According to the documentation :
After creating a device object, you must set the cooperative level for the device by using the IDirectSound8::SetCooperativeLevel method. Unless you do this, no sounds will be heard.
The problem is that I'm writing a console program, and SetCooperativeLevel requires a HWND as a first argument. I don't have any HWND to deal with in the console program. I tried providing a null pointer but it failed with a DSERR_INVALIDPARAM error code.
What HWND value should be provided to IDirectSound8::SetCooperativeLevel in a console program ? The audio part of the program is planned to be built as a shared library, so it has little to no knowledge of the "outside" program.
Thanks for any advice !
Note : I know that there is a somewhat better solution for simply rendering audio, like using SDL, OpenAL, SFML (based on OpenAL), but for my current project DirectSound is enforced.
Edit : I found a message from a Microsoft engineer that removes doubts about using the desktop window or the console window as a HWND for SetCooperativeLevel when creating GLOBAL_FOCUS buffers.
Although I have not tested this myself, you may have some success creating a hidden window and passing its HWND to the SetCooperativeLevel method. SetCooperativeLevel uses this hwnd to determine when your application has input focus; therefore, if you select a cooperative level where the input focus doesn't matter (eg, DSSCL_NORMAL), a hidden window (which will never receive input focus) should be fine.
you can use this ::GetDesktopWindow().

How do win32 apps using sigc and glib implements Message loop

If an Win32 app has its UI designed using sigc and glibmm,
how does it implement its message loops?
does it still use win32 API such as GetMessage, DispatchMessage, TranslateMessage, etc?
or they use other functions to finish this?
And the default WinProc is still there?
sigc+glibmm is the C++ level on top of the GTK C Framework callbacks/main loop which is on top of the native callback/main loop (NSRunLoop for MacOSX and GetMessage on Windows).
GetMessage must be called by every GUI application on Windows to get the absolute basics like a window handle, key presses and mouse movements.
TranslateMessage is not required because accelerator keys are handled with GTK's own implementation.
SendMessage is used very rare, most calls that require SendMessage are calls to a client control like a button or text field widget. In GTK they are implemented as GtkButton and GtkEntry and GTK can directly use the C implementation without going through the windows message dispatching.
All Windows GUI apps have to run a message pump based on GetMessage, TranslateMessage, DispatchMessage. Frameworks typically shield you from the implementation details, but somewhere in the framework will be a message pump.
The same is true for window procedures. Although you may never have to write one or interact with one, the framework will have to provide window procedures for top-level windows, and possibly for child windows depending on how the framework is implemented.

VB6 quitting while debugging with active Windows message hook

One of our largest old VB6 apps has some code in it to allow other apps (including some dotNET ones) to pass an ID to it via a Windows message - this ID is then used by the VB6 app to load an entry in a regular Windows form. The message hook is added after the user is logged in and authenticated, and removed once they logout.
Public Sub HookClaimFinderCall()
lpPrevWndProc = SetWindowLong(gHW, GWL_WNDPROC, AddressOf WindowProc)
End Sub
Public Sub UnhookClaimFinderCall()
Dim temp As Long
If gHW <> 0 Then temp = SetWindowLong(gHW, GWL_WNDPROC, lpPrevWndProc)
End Sub
Private Function WindowProc(ByVal hw As Long, ByVal uMsg As Long, ByVal wParam As Long, ByVal lParam As Long) As Long
If uMsg = WM_FINDCLAIM Then
MasterFindClaim lParam
End If
WindowProc = CallWindowProc(lpPrevWndProc, hw, uMsg, wParam, lParam)
End Function
However, there are two issues with this. The first relates to Visual Studio 6. If the code is being debugged, and an error occurs to bring up the "Continue End Debug Help" dialog box, pressing End quits Visual Studio instantly (losing any unsaved changes). This does not happen if the message hook has not been activated yet. What causes this, and is there anything I can do to stop it short of commenting out the code that loads the hook?
Secondly, if the users quit the app without logging out properly (by whatever means), what happens to the message hook?
I hope I've got all the terms right in the above...
You are going to get crashing (or disappearing VB6) if you initiate a global hook and attempt to Debug in VB6. VB6 is kind of like a simulator, it doesn't exactly duplicate the runtime of a VB6 application and hooking is one of the areas it fails miserably at (though it can't really be blamed if you understand what is going on). For all of the global hooks we use in our applications we check to see if VB6 is running in IDE mode (there are several ways to do this) and if it is, don't run the global hook. If you absolutely have to run the global hook, do not stop the application in the debugger - use debug.print or some other means, but don't stop the application otherwise you will have milliseconds to seconds before it "goes away".
Although you should unhook before exiting the application, when the message pump hits the hook and the handle of the application where the hook takes place no longer exists, it is a fairly cheep operation to ignore that hook. Now if you ran the application and exited thousands of times, it would probably build up, but that I think is the least of your concerns.
Kris's answer is correct. A couple of additional points.
Strictly this is subclassing not message hooking.
It's also good practice in your WindProc to detect WM_NCDESTROY and unhook when that message is received. The message means the window is about to be destroyed - it should be received if the user quits the app, no matter how they do it.
There are some API calls in Windows 2000 and later that make it easier to manage subclassing. As always, Karl Peterson has an excellent article with excellent VB6 code here.

Detecting Notification Balloons

Using WinXP. What I need to do (pref in VB or c#) is to detect when another (closed source) program displays a notification balloon in the tray - and grab the details. Any help would be appreciated. Thanks
In similar situations, I have used the Microsoft tool Spy++ to grab the window information and then uses pinvoke calls to FindWindow to detect when the window is present.
I've not tried with a notification balloon, but I imagine that a pinvoke call to GetText would retrieve the contents.
I think you'll need to use pinvoke to do this from a .net language.
On the system I'm using now (Vista Business SP2), balloon windows always seem to have window class #32769 (reserved for desktop windows) and the windows style bit TTS_BALLOON set.
The following might work: Determine the parent window for all notification balloons by creating a temporary one, getting its hWnd, and calling GetParent() before deleting it. You could then periodically poll the children of this parent hwnd (using EnumWindows() or FindWindowEx()) looking for windows with the required class and style.
This seems highly non-portable to me, and likely to require a lot of testing on a variety of platforms.
pinvoke.net and spy++ might be useful.
Good luck!
You will definitely need to use Win API calls to achieve this. If this is the only thing you're trying to do, you'd be better off using straight C or C++ so you don't have to do a bunch of platform invoke for C# or VB.
Since andyjohnson identified that the window class for all notification balloons is #32769, and that they have the TTS_BALLOON style set, you could use a CBT hook (if you're not familiar with Win32 hooks, you might want to read up on them), to get a callback whenever a window is created, and check for windows of that class and with that style.
I'm not sure, though, if a new balloon window is created for second and subsequent popups or if the same one is just hidden and reshown. If this is the case, you might need a CallWndProc hook, to get WM_SHOWWINDOW messages.
Edit:
I should mention that the hooks that I've mentioned cannot be implemented in .NET. Except for the low-level keyboard and mouse hooks, global system hooks must be implemented in a native (unmanaged) DLL. Windows will load this DLL into other processes, and if a managed DLL gets loaded into a process that doesn't have the .NET CLR loaded, it will crash that process. (Even if the CLR is loaded, it might be at a different address, also causing a crash.)
So you must build your hooks in a native (unmanaged) DLL. It's possible to interface from here to a managed application, such as Michael Kennedy has done on Code Project, but to do it properly, and handle the hook types I've mentioned above, you'd need to use interprocess communication, a step that Michael Kennedy left out. All in all, for the purpose you've described, it would probably be easier to just build the whole thing in native code.

Resources