In a Win32 C++ application, we start a message loop that fetches messages from a queue, translates them and then dispatches them. Eventually, each message reaches our WndProc where the associated event can be handled.
I understand that part. What I don't understand is the in between goings-on. Specifically:
Different kinds of OS interrupt handlers must be placing messages in the said 'message queue', but where within the process address space does this queue reside? How is it exposed to the interrupt handler code?
What does it mean to 'translate' the message? What does the call to TranslateMessage() really do?
Once dispatched by DispatchMessage(), what all places does the message swing by before reaching my WndProc (i.e. what does the OS do with it)?
If anyone knows the answers to the above, kindly satisfy my curiosity. Thanks.
The OS maintains a message queue, where it puts the events (e.g., from interrupts or other sources). It then sends the messages from that queue to all windows, depending on the message (e.g., it won't send key messages to a window that doesn't have focus).
Applications can have their own queue to process messages. Those queues are created on request (only if needed).
Translating a message is used to create messages that are not 'real' events. For example, the WM_CONTEXTMENU message is 'translated' from either a mouse right-click, or the context menu key, or shift-F10. The WM_CHAR is translated from WM_KEYDOWN messages. And of course many other messages are 'translated' that way.
A message is posted to every window that should receive it. The OS decides depending on the type of message whether a window should receive that message or not. Most messages are waited for by the system, i.e., the message won't get posted to another window until it was processed by the window. This has a great impact for broadcast messages: if one window doesn't return when handling that message, the queue is blocked and other windows won't receive the message anymore.
It depends on how your message is sent and how it's handled.
When you call SendMessage, if the target window is owned by the current thread, the call bypasses the message queue for the window and the window manager directly calls the windowproc on the target window. If the target window is owned by another thread, the window manager effectively calls PostMessage and pumps window messages until the target window returns from the window proc.
When you call PostMessage, the window manager marshals the message parameters and inserts the corresponding object onto the message queue for the target window. When it next calls GetMessage the message is removed from the message queue.
The window manager also registers for raw input events from the input devices (keyboard and/or mouse) and it generates messages for those input events. It then inserts those messages in the queue as appropriate (the processing of input events is complicated because it depends on what messages are already in the message queue for the window).
As Stefan indicated, TranslateMessage just translates accelerator keys - for instance it converts key sequences to WM_COMMAND messages.
Different kinds of OS interrupt handlers must be placing messages in the said 'message queue', but where within the process address space does this queue reside? How is it exposed to the interrupt handler code?
Windows are associated with threads. Each thread with a window has a thread queue in the process's address space. The OS has an internal queue in its own address space for the hardware-generated events. Using details of the event and other state information (e.g., which window has the focus), the OS translates the hardware events into messages that are then placed in the appropriate thread queue.
Messages that are posted are placed directly in the thread queue for the target window.
Messages that are sent are usually processed directly (bypassing the queue).
The details get hairy. For example, the thread queues are more than lists of messages--they also maintain some state information. Some messages (like WM_PAINT) aren't really queued, but synthesized from the additional state information when you query the queue and it's empty. Messages sent to windows owned by other threads are actually posted to the receiver's queue rather than being processed directly, but the system makes it appear like a regular blocking send from the caller's point of view. Hilarity ensues if this can cause deadlock (because of circular sends back to the original thread).
The Jeffrey Richter books have a lot (all?) of the gory details. My edition is old (Advanced Windows). The current edition seems to be called Windows via C/C++.
The OS does a LOT of work to make the message stream appear rational (and relatively simple) to the caller.
What does it mean to 'translate' the message? What does the call to TranslateMessage() really do?
It watches for virtual key messages and, when it recognizes a key-down/key-up combination, it adds character messages. If you don't call TranslateMessage, you won't receive character messages like WM_CHAR.
I suspect it sends the character message directly before returning (as opposed to posting them). I've never checked, but I seem to recall that the WM_CHAR messages arrive just before the WM_KEYUP.
Once dispatched by DispatchMessage(), what all places does the message swing by before reaching my WndProc (i.e. what does the OS do with it)?
DispatchMessage passes the message to the WndProc for the target window. Along the way, it some hooks may get a chance to see the message (and possibly interfere with it).
To address the last subquestion, a dispatched message will go to your WindowProc after it's been piped through all hooks (WH_CALLWNDPROC)
Not absolutely positive about this but my best guess says:
The queue is a system object that you access with Win32 API calls. It is not in your process address space at all. So the interrupt handlers can access it (probably through the HAL (Hardware Abstraction Layer) of the kernel).
In Win16, that call took the various subparts of a bigger message and mashed them into a whole. So TranslateMessage would add the WM_KEYPRESS when it found the corresponding WM_KEYDOWN WM_KEYUP sequence. It would also turn various button click messages into doubleclick messages based on internal setting and the timestamps of the messages. Whether it still does this in Win32, I don't know.
DispatchMessage is probably where Window message hooks get processed. So if there's a hook on your window, it is either called here or when the GetMessage is called. I'm not sure. Other than that, DispatchMessage just looks up the WndProc address associated with the window and calls it. There's not much else for it to do.
Hope that helps.
Related
I'd like to confirm what I think to be true. When I use Windows SendMessage(), this is a deterministic call in that it will execute immediately (will not return until the message is processed) as opposed to PostMessage() which is non-deterministic as it can be preempted by any other message that happens to be in the queue at that moment in time (in fact it will not be executed until it hits the message loop).
Is this a fair assessment, or am I missing something?
That is essentially true for in-process calls. Cross-process SendMessage works similar, but processing of the message doesn't begin until the receiver process calls GetMessage (or its kin).
Your UI thread has a message pump that looks something like:
while (GetMessage(&msg))
DispatchMessage(&msg);
PostMessage causes the message to be put onto the message queue. GetMessage removes the oldest message from the queue (FIFO*).
DispatchMessage causes the WndProc associated with the message's target window to be called with the message.
SendMessage basically bypasses this chain and calls the WndProc directly (more or less).
A lot of standard window messages result in a chain of SendMessage calls where sending one message sends another which sends another. That chain is often referred to as "the current dispatch". If you need your message to be processed inside the chain, use SendMessage. If you need it to be processed after the current dispatch has completed, use PostMessage.
You can use a tool like Spy++ to see Windows messaging in action, or to debug problems you're having with message order of operations.
[*] It's not strictly a FIFO queue because certain kinds of messages (i.e. timer, mouse, keyboard) are not actually posted to the queue, but rather generated on the fly. For the sake of simplicity, you can think of it as a FIFO.
I read some stuff on Win32 and how the message loop works, and there's something that is still unclear to me: What exactly is stored in the message queue? The integer value that corresponds to a message (WM_COMMAND, WM_CREATE, etc) or a pointer to a MSG structure containing the message integer value and other stuff like wParam, lParam, etc?
To answer your question narrowly, each message in the queue stores, at the least,
a window handle to which the message is directed,
the message code, wParam and lParam, as you already correctly noted,
the time when the message was posted, that you retrieve with GetMessageTime(),
for UI messages, the position of the cursor when the message was posted (see GetMessagePos()).
Note that not all messages are actually stored in the queue. Messages that are sent with SendMessage() to a window from the thread that owns the window are never stored; instead, a receiver window's message function is called directly. Messages sent from other threads are stored until processed, and the sending thread blocks until the message is replied to, either by exiting the window function or explicitly with a call to ReplyMessage(). The API function InSendMessage() helps figure out whether the windows function is processing a message sent from another thread.
Messages that you or the system post are stored in the queue, with some exceptions.
WM_TIMER messages are never actually stored; instead, GetMessage() constructs a timer message if there are no other messages in the queue and a timer has matured. This means that, first, the timer messages have the lowest dequeuing priority, and, second, that multiple messages from a short period timer would never overflow queue, even if GetMessage() is not called for a while. As a result, a single WM_TIMER message is sent for the given timer, even if the timer has elapsed multiple times since the last WM_TIMER message from that timer has been processed.
Similarly, WM_QUIT is also not stored, and rather only flagged. GetMessage() pretends to have retrieved the WM_QUIT after the queue has been exhausted, and this is the last message it retrieves.
Another example is the WM_PAINT message (hat tip to #cody-gray for reminding about this). This message is also simulated when any part of the window¹ is marked as "dirty" and needs repainting. This is also a low-priority message, made so that multiple invalidated regions in a window are repainted all at once when the queue becomes empty, to reduce responsiveness of the GUI and reduce flicker. You can force an immediate repaint by calling UpdateWindow(). This function acts like SendMessage(), in the sense that it does not return until the exposed part of the window is redrawn. This function does not send a WM_PAINT to the window if the invalid region of that window is empty, as an obvious optimization.
Probably, there are other exceptions and internal optimizations.
Messages posted with PostMessage() end up in the queue of a thread that owns the window to which the message is posted.
In what form the messages are stored internally, we do not know, and we do not care. Windows API abstracts that completely. The MSG structure is filled in memory you pass to GetMessage() or PeekMessage(). You do not need to know or to worry about the details of internal implementation beyond those documented in Windows SDK guides.
¹ I do not know how exactly WM_PAINT and WM_TIMER are prioritized relative to each other. I assume WM_PAINT has a lower priority, but I may be wrong.
The message queue in Windows is an abstraction. Very helpful to think of it as a queue but the actual implementation of it is far more detailed. There are four distinct sources of messages in Windows:
Messages that are delivered by SendMessage(). Windows directly calls the window procedure, the message isn't returned by Peek/GetMessage() but does require a call to that function to get dispatched. By far the most messages are delivered this way. WM_COMMAND is like that, it is directly sent by the code that translates a key-down event, like TranslateAccelerator(). No queue-like behavior.
Messages that are synthesized from the window state. Best examples are WM_PAINT, delivered when the "window has a dirty rectangle" state flag is set. And WM_TIMER, delivered when the "timer has expired" state flag is set. These are 'low priority' messages, only delivered when the message queue is empty. They are delivered by GetMessage() but do not otherwise live on the queue.
Input event messages for the keyboard and mouse. These are the messages that truly have queue-like behavior. For the keyboard this ensures that type-ahead works, no keystroke gets lost when the program isn't ready to accept a keystroke. A bunch of state is associated with it, the entire state of the keyboard gets copied for example. Roughly the same for the mouse except that there's less state. The WM_MOUSEMOVE message is an interesting corner case, the queue does not store every single pixel traversed by the cursor. Position changes are accumulated into a single message, stored or delivered when necessary.
Messages stored in the queue by an explicit PostMessage() call. That's entirely up to the program code, clearly it only needs to store the arguments of the call plus the time the call was made so it can accurately be replayed back at GetMessage() time.
MSDN has a nice article here explaining everything on messages and message queues.
But to answer your question, the Queue exists for each window, it temporarily stores the messages and their associate parameters, whether of not its a queue of MSG's is implementation defined, but it most likely is (or something similar). It should also be noted that not all messages will go to the queue, some require immediate processing.
The general question, is if I post several messages to the windows message pump from a separate worker thread, will they appear at their destination in the order I sent? ie..
::PostMessage(m_hUsers, WM_BULKPROCESS, 0, 0);
// ... some processing here ...
::PostMessage(m_hUsers, WM_BULKDONE, 0, 0);
m_hUsers is a handle (HWND) to a window I'm sending the messages to from my worker thread. So, will WM_BULKPROCESS always show up first in the window (and therefore be processed by the handler in that dialog class), or is it possible for them to get out of order, ie WM_BULKDONE gets processed before WM_BULKPROCESS, even though it was sent last?
There are a few exceptions (like WM_PAINT), but generally, the order of messages is kept.
Imaging trying to make sense of mouse input if messages appeared in the wrong order!
Quote from GetMessage
During this call, the system delivers pending, nonqueued messages,
that is, messages sent to windows owned by the calling thread using
the SendMessage, SendMessageCallback, SendMessageTimeout, or
SendNotifyMessage function. Then the first queued message that matches
the specified filter is retrieved. The system may also process
internal events. If no filter is specified, messages are processed in
the following order:
Sent messages
Posted messages
Input (hardware) messages and system internal events
Sent messages (again)
WM_PAINT messages
WM_TIMER messages
Window messages are stored in a queue. So you can rely on FIFO mechanism.
They should be unless you have code in the message pump that specifically dispatch the messages (either intentionally or not) differently e.g. somehow pick two messages and dispatch them out of order. Normally programmers call DispatchMessage for each message in the order you get from the queue.
I suspect the issue is synchronization and not the message queue. If your code allows multitple invocations of the worker thread proc you have to manage this more tightly to know which "instance" of the worker thread is posting the messages.
Have you checked to make sure you only one worker thread executing at a time, or that the m_hUsers window handle is protected from being changed between BULKPROCESS and BULKDONE?
SendMessage can be useful for managing the BULKDONE because it will block until the message has been processed allowing the code invoking the worker thread to synchronize the invokation of worker threads and truely know that one worker thread has finsished before invoking another. Postmessage will not block but remember the time sensitive part of your worker thread is
`// ... some processing here ...
not sending the windows messages.
Simple task: Send a windows message with dynamically allocated data, e.g. an arbitrary length string. How would you manage the responsibility to free this data?
The receiver(s) of the windows message could be responsible to free this data. But: How can you guarantee that all messages will actually be received and thus the linked data will be freed? Imagine the situation that the receiver is shutting down, so it won't process it's message queue any more. However, the message queue still exists (for some time) and can still accept messages, which won't be processed any more.
Thanks!
PostMessage returns a BOOL that tells you whether the message was posted or not. This is usually good enough, because your window should be valid until it receives the WM_DESTROY and the WM_NCDESTROY messages. After a call to DestroyWindow (which sends these messages) you should not be able to successfully call PostMessage again.
Now, if your PostMessage returns FALSE you have to clean up. If it doesn't, the window procedure has to clean up. Don't send messages that have to be cleaned up to random windows that might not handle them. Actually, don't send any WM_USER + x messages to any windows you don't handle.
There's nothing to do here. As soon as the call to SendMessage returns, you can free the data. As it happens, the other app isn't looking at your memory anyway since it's in a different process. Instead Windows marshals the data across the process boundary.
What's more, if you are receiving the data in a WndProc, you can't take a copy of the pointer to the string. Instead you must take a copy of the contents of the string since that pointer is only valid for the duration of that call to WndProc.
The other point to make is that you have a confusion about the message queue. When you send a message, that happens synchronously and the queue is not involved. The message queue is where posted messages are placed. They are process asynchronously.
Windows messages seems a good way to notify an application on Windows OSes. It actually works well, but few question comes up to my mind:
How to specify structured data to the lparam of the SendMessage routines (like many message codes does)? I mean... of course the parameter is a pointer, but how the process access to it? Maybe is it allocated by a DLL loaded by the processes sending/receiving the message?
Is it possible to share message structured parameters (between sender and receiver)? They are marshalled between the send operation and the peeking operation? If this is the case, it is possible to return data from the caller by modifying the structured parameter? This could be usefull with SendMessage, since it is executed synchronously, instead the PostMessage routine.
Other doubts...
What the differences from PostMessage and SendNotifyMessage?
Is it possible to cause a deadlock in the case an application calls SendMessage to itself while processing the message pump?
If the message is one of the standard window's messages - usually with a message id between 0 and WM_USER, then the systems window message dispatching logic contains code to marshal the struct to the any processes the message is dispatched to.
Messages above WM_USER get no such treatment - and this includes all the common control messages introduced with Windows 95 - you cannot end any of the LVM_* (list view messages) or other new control messages to a control in a different process and get a result back.
WM_COPYDATA was specifically introduced as a general purpose mechanism for user code to marshal arbitrary data between processes - outside of WM_COPYDATA (or reusing other windows standard messages) there is no way to get windows to automatically marshal structured data using the message queue mechanism into another process.
If it is your own code doing the sending AND receiving of messages, you could use a dll to define a shared memory section, instead of sending pointers (the dll might be based differently in each process) sends offsets to the shared memory block.
If you want to exchange structured data with external applications that do not marshal their data (for example to extract data from a list or tree view) then you need to perform dll injection so you can send and process the message from "in-process".
SendNofityMessage is different to PostMessage because PostMessage always puts the message in the message queue, whereas SendNotifyMessage acts like SendMessage for windows in the same process. Then, even if the target window is in another process, the message is dispatched DIRECTLY to the window proc not placed in the posted message queue for retreivel via GetMessage or PeekMessage.
Lastly it is possible to cause a deadlock - however while in a "blocking" sendmessage waiting for another thread to reply, SendMessage will dispatch messages sent (not posted) from other threads - to prevent deadlocks. This mitigates most potential deadlocks, but its still possible to create deadlocks by calling other blocking apis, or going into modal message processing loops.
Your concerns apply primarily in the case of messages sent between processes -- within a process, you can just send a pointer, and the sender just has to ensure the data remains valid until the receiver is finished using it.
Some interprocess messages require you to work with an HGLOBAL (e.g., the clipboard messages). Others require an explicit block size (e.g., with WM_COPYDATA). Still others send data in a pre-specified structure (e.g., a zero-terminated string) to support marshalling.
Generally speaking you can't return a value by modifying the block you received. To return a value, you need to (for example) send a reply message.
SendMessage has some (fairly complex) logic to prevent deadlocks, but you still need to be careful.