I'm using WinHTTP in sync mode, without passing the WINHTTP_FLAG_ASYNC flag, and I thought that the callback is always being called synchronously. That is indeed what's happening most of the time, but sometimes, when calling WinHttpCloseHandle, the callback isn't called with the WINHTTP_CALLBACK_STATUS_HANDLE_CLOSING notification right away. Instead, it's being called afterwards from a different thread.
It that expected behavior? Why does it become async for some cases, if the seesion is sync? I know how to fix it (waiting for the WINHTTP_CALLBACK_STATUS_HANDLE_CLOSING notification if I don't get it right away), but I don't understand why that's the behavior that I'm seeing.
WinHTTP does not promise synchronous "same thread" callbacks in synchronous mode. On the contrary, MSDN states the opposite:
The callback function must be threadsafe and reentrant because it can be called on another thread for a separate request, and reentered on the same thread for the current request. It must therefore be coded to handle reentrance safely while processing. When the dwInternetStatus parameter is equal to WINHTTP_CALLBACK_STATUS_HANDLE_CLOSING, the callback does not need to be able to handle reentrance for the same request, because this callback is guaranteed to be the last, and does not occur when other messages for this request are handled.
This means that the symptom you are seeing is basically behavior by design and is not related to async mode: some callback calls might be sent to you from worker threads and then thread racing might reach your code late in your callback. You need to take this into consideration and either ignore those late calls, or synchronize with them, or reset callbacks early enough explicitly to not receive late notifications.
Regarding WINHTTP_CALLBACK_STATUS_HANDLE_CLOSING specifically MSDN explains what you can rely on exactly (see quote above).
Related
I don't understand part of the latest Windows threadpool API. I need help with that.
From the documentation, the recipe to use it for I/O (in my case, for SOCKET) can be summarized as follows:
Call CreateThreadpoolIo.
Call StartThreadpoolIo. You can find this warning there:
You must call this function before initiating each asynchronous I/O operation on the file handle bound to the I/O completion object. Failure to do so will cause the thread pool to ignore an I/O operation when it completes and will cause memory corruption.
Call the operation on the file handle (e.g., WSARecvFrom). If it fails, call CancelThreadpoolIo. Otherwise, process the result when it is available. WSARecvFrom, when used asynchronously, asks for a WSAOVERLAPPED (that you have to create beforehand) but not for any information that links it to the previous call to StartThreadpoolIo. CancelThreadpoolIo only asks for the PTP_IO, but not for any additional information to derive a specific asynchronous operation.
Repeat steps 2 and 3.
Call CloseThreadpoolIo to finish. You can find this warning there:
It may be necessary to cancel threadpool I/O notifications to prevent memory leaks. For more information, see CancelThreadpoolIo.
I usually need it for UDP, so I strive to have several reception operations queued (asynchronous WSARecvFrom operations started) at any given time. That way I don't have to rush to start another reception operation at the beginning of the callback function nor synchronize access to the reception buffers (I can have a pool of them, each one able to contain a datagram, and reissue the reception operation when I finish processing each message; in the interim, other queued operations will keep the receiver busy). Datagrams are independent and self contained. I'm aware that this approach may not be valid for TCP.
StartThreadpoolIo/CancelThreadpoolIo seem to me the source of the problem: StartThreadpoolIo and WSARecvFrom are not directly bound (they don't share any arguments). So:
How can the framework know which operation to cancel when you call CancelThreadpoolIo? How does it cancel just the operation that failed and not any of the pending ones?
You can say, "don't call StartThreadpoolIo concurrently". I can live without several concurrent WSARecvFrom's, but I can't live without concurrent WSARecvFrom and WSASendTo. So I think being unable to have several asynchronous operations at the same time can't be the way the API was designed.
You can say, "call StartThreadpoolIo only once, that will suffice to register the callback; it is an on/off process". But the documentation says:
You must call this function before initiating each asynchronous I/O operation on the file handle...
You can say, "it cancels the operation started by the same thread that just called StartThreadpoolIo". But then the advice of calling CancelThreadpoolIo in the context of calling CloseThreadpoolIo doesn't make sense (I will call CloseThreadpoolIo from the thread that triggers stopping, which will be completely independent from the threads issuing the asynchronous operations; and a single call to CancelThreadpoolIo may not be enough to cancel several operations). Being unable to trigger cancellation from a different thread is a serious limitation, anyway. I'm aware of the existence of CreateThreadpoolCleanupGroup, but my question is more fundamental. I want to understand how this API can be fundamentally right and useful.
You can say "call CreateThreadpoolIo several times, so that you have independent PTP_IO's to work with". It doesn't work. When I call CreateThreadpoolIo a second time, nullptr is returned.
Am I wrong, or is this API awkward? Normally, other asynchronous APIs work with one of these patterns:
Create an operation and receive a handle => call methods passing the handle.
Create a reusable handle => call methods (including starting operations) passing the handle.
The latest Windows threadpool API, in which the handle seems to be implicit, or there are several handles for the same operation (TP_IO, WSAOVERLAPPED, StartThreadpoolIo) and they aren't all explicitly linked together, uses neither of them.
Thank you very much for your help.
How can the framework know which operation to cancel when you call CancelThreadpoolIo? How does it cancel just the operation that failed
and not any of the pending ones?
CancelThreadpoolIo() doesn't cancel IO. It is reciprocal to StartThreadpoolIo(). StartThreadpoolIo() prepares threadpool to accept a completion. If threadpool doesn't expect a completion, it won't wait for it, thus you may miss it. If threadpool expects a completion but completion doesn't happen, threadpool may waste resources.
CancelThreadpoolIo() undoes whatever StartThreadpoolIo() did.
An asynchronous call to ReadDirectoryChangesW() can be cancelled by the CancelIo() function. However, by the time CancelIo() has been called, the notification buffer associated with ReadDirectoryChangesW() may be half filled in. The question is, what happens with those notifications? Should they be processed in a normal way?
More specifically, I issued an overlapped ReadDirectoryChangesW() command with a completion routine, and than cancelled it by means of CancelIo(). When my completion routine is called with an ERROR_OPERATION_ABORTED error, should I still check the notification buffer for possible notifications?
Clarification:
My File System Listener component successfully serves for my company already more than ten years. Now I'm going to modify the component in order to implement a more sophisticated monitoring policy. With this policy, cancelling a particular ReadDirectoryChangesW() request doesn't mean cancelling of monitoring, and I do not want to miss even a single notification.
OK, I found experimentally that the second parameter of completion routine (dwNumberOfBytesTransfered) in the case of ReadDirecoryChangesW() function contains number of bytes written to notification buffer. Unfortunately, I didn't find a clear confirmation of that in documentation.
However, this hypothesis seems reasonable, and if it is always true, I can conclude about presence of notifications by checking of that parameter regardless of the completion routine's first parameter (dwErrorCode). This solves the problem.
I remember there was a way to do this, something similar to unix signals, but not so widely used. But can't remember the term. No events/mutexes are used: the thread is just interrupted at random place, the function is called and when it returns, the thread continues.
Windows has Asynchronous Procedure Calls which can call a function in the context of a specific thread. APC's do not just interrupt a thread at a random place (that would be dangerous - the thread could be in the middle of writing to a file or obtaining a lock or in Kernel mode). Instead an APC will be dispatched when the calling thread enters an alterable wait by calling a specific function (See the APC documentation).
If the reason that you need to call code in a specific thread is because you are interacting with the user interface, it would be more direct to send or post a window message to the window handle that you want to update. Window messages are always processed in the thread that created the window.
you can search RtlRemoteCall, it's an undocumented routine though. there's APC in Windows semantically similar to Unix signal, however APC requires target thread is in an alertable state to get delivered, it's not guaranteed this condition is always met
I've been reading the MSDN documentation for IcmpSendEcho2 and it raises more questions than it answers.
I'm familiar with asynchronous callbacks from other Win32 APIs such as ReadFileEx... I provide a buffer which I guarantee will be reserved for the driver's use until the operation completes with any result other than IO_PENDING, I get my callback in case of either success or failure (and call GetCompletionStatus to find out which). Timeouts are my responsibility and I can call CancelIo to abort processing, but the buffer is still reserved until the driver cancels the operation and calls my completion routine with a status of CANCELLED. And there's an OVERLAPPED structure which uniquely identifies the request through all of this.
IcmpSendEcho2 doesn't use an OVERLAPPED context structure for asynchronous requests. And the documentation is unclear excessively minimalist about what happens if the ping times out or fails (failure would be lack of a network connection, a missing ARP entry for local peers, ICMP destination unreachable response from an intervening router for remote peers, etc).
Does anyone know whether the callback occurs on timeout and/or failure? And especially, if no response comes, can I reuse the buffer for another call to IcmpSendEcho2 or is it forever reserved in case a reply comes in late?
I'm wanting to use this function from a Win32 service, which means I have to get the error-handling cases right and I can't just leak buffers (or if the API does leak buffers, I have to use a helper process so I have a way to abandon requests).
There's also an ugly incompatibility in the way the callback is made. It looks like the first parameter is consistent between the two signatures, so I should be able to use the newer PIO_APC_ROUTINE as long as I only use the second parameter if an OS version check returns Vista or newer? Although MSDN says "don't do a Windows version check", it seems like I need to, because the set of versions with the new argument aren't the same as the set of versions where the function exists in iphlpapi.dll.
Pointers to additional documentation or working code which uses this function and an APC would be much appreciated.
Please also let me know if this is completely the wrong approach -- i.e. if either using raw sockets or some combination of IcmpCreateFile+WriteFileEx+ReadFileEx would be more robust.
I use IcmpSendEcho2 with an event, not a callback, but I think the flow is the same in both cases. IcmpSendEcho2 uses NtDeviceIoControlFile internally. It detects some ICMP-related errors early on and returns them as error codes in the 12xx range. If (and only if) IcmpSendEcho2 returns ERROR_IO_PENDING, it will eventually call the callback and/or set the event, regardless of whether the ping succeeds, fails or times out. Any buffers you pass in must be preserved until then, but can be reused afterwards.
As for the version check, you can avoid it at a slight cost by using an event with RegisterWaitForSingleObject instead of an APC callback.
What happens when you call WaitForSingleObject() on a handle you've created with CreateFile() or _get_osfhandle()?
For reasons not worth explaining I would like to use WaitForSingleObject() to wait on a HANDLE that I've created with _get_osfhandle(fd), where fd comes from a regular call to _open(). Is this possible?
I have tried it in practice, and on some machines it works as expected (the HANDLE is always in the signaled state because you can read more data from it), and on some machines WaitForSingleObject() will block indefinitely if you let it.
The MSDN page for WaitForSingleObject() says that the only supported things that it handles are "change notifications, console input, events, memory resource notifications, mutex, processes, semaphores, threads, and waitable timers."
Additionally, would it be different if I used CreateFile() instead of _get_osfhandle() on a CRT file descriptor?
Don't do it. As you can see, it has undefined behavior.
Even when the behavior is defined, it's defined in such a way as to be relatively not useful unless you don't like writing additional code. It is signaled when any asynchronous I/O operation on that handle completes, which does not generalize to tracking which I/O operation finished.
Why are you trying to wait on a file handle? Clearly the intent matters when you are doing something that isn't even supported well enough to not block indefinitely.
I found the following links. The concensus seems to me, don't do it.
Asynch IO explorer
Waiting on a file handle
When an I/O operation is started on an
asynchronous handle, the handle goes
into a non-signaled state. Therefore,
when used in the context of a
WaitForSingleObject or
WaitForMultipleObjects operation, the
file handle will become signaled when
the I/O operation completes. However,
Microsoft actively discourages this
technique; it does not generalize if
there exists more than one pending I/O
operation; the handle would become
signaled if any I/O operation
completed. Therefore, although this
technique is feasible, it is not
considered best practice.
Egghead Cafe:
Use ReadDirectoryChangesW in
overlapped mode. WaitForSingleObject
can wait on the event in the
OVERLAPPED struct.
You can also use the API
WaitForSingleObject() to wait on a
file change if you use the following
change notification function:
FindFirstChangeNotification()
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/findfirstchangenotification.asp
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/waitforsingleobject.asp
An interesting note on "evilness" of ReadDirectoryChangesW:
http://blogs.msdn.com/ericgu/archive/2005/10/07/478396.aspx