While sending a bio down for write IO, I acquire a semaphore (down_interruptible). And in the IO completion routine i.e. inside bio_end_io, I release the semaphore (up).
I know that acquiring a semaphore in completion routine is illegal as the blocking operations are not allowed in atomic context. But does the same restriction apply while releasing the semaphore?
Related
Trying to understand how this works... do I have to create various threads to take advantage of the functionality for GetOverlappedResultEx? However why couldn't I just put GetOverlappedResult in a separate thread from the main thread to handle blocking of the IO and not interfere with main operations?
GetOverlappedResult function
https://learn.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-getoverlappedresult
Retrieves the results of an overlapped operation on the specified file, named pipe, or communications device. To specify a timeout interval or wait on an alertable thread, use GetOverlappedResultEx.
https://learn.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-getoverlappedresultex
Retrieves the results of an overlapped operation on the specified file, named pipe, or communications device within the specified time-out interval. The calling thread can perform an alertable wait.
https://learn.microsoft.com/en-us/windows/win32/fileio/alertable-i-o
You handle threads, for concurrency, yourself.
There are basically three ways to do it:
Having initiated an overlapped (i.e., async completion) I/O operation you do something else and then every once in awhile poll the handle to see if the overlapped operation has completed. This is how you can use GetOverlappedResult looking for STATUS_PENDING to see if the operation isn't done yet.
You sit around waiting for an overlapped operation to complete. But it's not as bad as that, because you can actually sit around waiting for any of a set of overlapped operations to complete. As soon as any one completes you handle it, and then loop around to wait for the rest. Handling it, of course, may fire off another asynch operation, you add that handle to the list. This is where you use WaitForSingleObject{Ex} or better WaitForMultipleObjects{Ex}.
You use I/O Completion ports. Here you pass some handles to a kernel object called an I/O Completion port - this kernel object cleverly combines a thread pool (that it manages itself) with callbacks. It is a very efficient way of dealing with multiple - in fact, very many - async operations in-flight simultaneously. In these callbacks you can do whatever you want, including initiating more async operations and adding them to the same I/O Completion port.
There is also a fourth concept: alertable I/O, which executes a callback on an "APC" on your thread that initiated the I/O, provided your thread is in an "alertable" state - which means it is executing one or another of certain APIs that wait in the kernel. But I've never used it, as it seems to have drawbacks (such as only working on the thread that initiated the I/O, and that the environment the callback environment runs in isn't as clear as it could be) and if you're going to go that far just figure out I/O Completion ports and use them.
Options #2 and #3 of course involve concurrent programming - so in both cases you have to make sure your callbacks are thread-safe with respect to your other threads.
There are plenty of examples of all these methods out there on the intertubes.
I know that the next async_write()'s should be performed when the previous one finished (with or without errors, but when it finished).
I would like to know what happens when, while making async_write() calls, if one of these takes long time for some reason or even never ends (I assume there is no timeouts here like in synchronous operations). When this operation will be considered as failed? When that operation that never ends is finally removed by the OS internally?
Maybe, are there timeouts involved and my assumptions are wrong?
I mean, the write operation is sent to the OS and could possibly block, indefinitely?
So the handler is never called and the next async_write()'s are never called.
NOTE: I am assuming that we are calling run() in several threads but the write operations should be sent in order so I am also assuming that the write handlers are wrapped with a strand.
Thank you for your time.
There are no explicit timeouts for asynchronous operations, but they can be cancelled through the IO object's cancel() member function. These operations will be considered as having failed only when the underlying OS call itself fails in a manner where a retry cannot reasonable occur. For example, if the write fails from:
EINTR, then the write will immediately be reattempted.
EWOULDBLOCK, EAGAIN, or ERROR_RETRY, then Boost.Asio will push the operation back into the job queue. This could occur if the write buffer was full, so pushing the operation back into the queue defers its reattempt, allowing other operations to be attempted.
Other errors will cause the operation to fail.
There should not be an indefinitely block in the system call. Boost.Asio sets the underlying IO objects to non-blocking, and provides synchronous blocking writes behavior by waiting on the associated file descriptor if a write failed with EWOULDBLOCK, EAGAIN, or ERROR_RETRY.
A strand is not affected by long term asynchronous operations. Strands are used to provide strict sequential invocation of handlers, not the operations themselves. In the case of composed operations, such as boost::asio::async_write, the intermediate handlers will also be invoked through the same strand as the final handler. Overall, this behavior helps provide thread safety, as:
All async_write_some operations initiated from intermediate handlers are within the strand.
The operation itself is not within the strand. This allows other for other handlers to run while the actual write is occurring.
The user handler will be invoked within the strand.
This answer may provide some more insight into composed operations and strands.
CancelIo() is supposed to cancel all pending I/O operations associated with the calling thread. In my experience, CancelIo() sometimes cancels future I/O operations as well. Given:
ReadFile(port, buffer, length, &bytesTransferred, overlapped);
If I invoke CancelIo(port) immediately before the read, GetQueuedCompletionStatus() will block forever, never receiving the read operation.
If I invoke CancelIo(port) immediately after the read, GetQueuedCompletionStatus() will return 0 with GetLastError()==ERROR_OPERATION_ABORTED
If I invoke CancelIo(port) and there are no pending or subsequent reads, GetQueuedCompletionStatus() will block forever.
The key point here is that there is no way to detect when CancelIo() has finished executing. How can I ensure that CancelIo() is done executing and it is safe to issue further read requests?
PS: Looking at http://osdir.com/ml/lib.boost.asio.user/2008-02/msg00074.html and http://www.boost.org/doc/libs/1_44_0/doc/html/boost_asio/using.html it sounds like CancelIo() is not really usable. Must customer requires Windows XP support. What are my options?
NOTE: I am reading from a serial port.
CancelIo() works fine. I misunderstood my code.
Upon further investigation it turns out that the code was invoking CancelIo() followed by ReadFile() with a timeout INFINITE. The completion port was never getting notified of the read because the remote end was never sending anything. In other words, CancelIo() did not cancel subsequent operations.
I found some eye-opening documentation here:
Be careful when coding for asynchronous I/O because the system reserves the right to make an operation synchronous if it needs to. Therefore, it is best if you write the program to correctly handle an I/O operation that may be completed either synchronously or asynchronously. The sample code demonstrates this consideration.
It turns out that device drivers may choose to treat an asynchronous operation in a synchronous manner if the data being read is already cached by the device driver. Upon further investigation, I discovered that when CancelIo() was being invoked before ReadFile() it would sometimes cause the latter to return synchronously. I have no idea why the completion port was never getting notified of ReadFile() after a CancelIo() but I can no longer reproduce this problem.
The completion port is signaled regardless of whether ReadFile() is synchronous or asynchronous.
Wait on (possibly with zero timeout) overlapped.Handle. It will be set whether the operation is completed or cancelled.
If you're already using overlapped operations, why do you need to cancel I/O at all? The entire concept of 'cancelling' an in-flight I/O operation is really race-prone, and totally subject to the underlying device stack you're trying to write to; really the only time you'd want to do this is to unblock another thread who is waiting on the completion of that I/O.
It is possible to write asynchronous I/O code without CancelIo function. The question depends on the scenario you are using CancelIO. Let's say that you need to implement file reading thread. Thread pseudo-code:
for(;;)
{
ReadFile(port, buffer, length, &bytesTransferred, overlapped);
WaitForMultipleObjects( overlapped event + stop event);
if ( stop event is signaled )
break;
if (overlapped event is signaled )
handle ReadFile results
}
Such thread reads file (socket, port etc.) using overlapped I/O. Most of the time it waits on WiatForMultipleObjects line. It wakes up when new data is available, or stop event is signaled. To stop this thread, set stop event from another thread. CancelIO is not used.
Why do we need Mutex and Events in Windows? In the sense couldn't windows have just Mutex? What is that can be done with Events that cannot be done with Mutex?
Events allows threads to block until some event (hence the name) is broadcast. Blocking on an Event means "Wake me when something happened"; you expect to be put to sleep. Event's are a signalling mechanism and provides support for this not found on mutexes, such as automatically being able to clear the signal as soon as someone who waited on it was woken up. Also, the API allows for blocking until one of or all of several events are signalled.
The mutex (Mutual Exclusion), on the other hand, is a scoped coordination mechanism for shared resources. Think transaction. You're not expecting to wait but want to access some shared resource, and only in the event that others are already accessing it, you're blocking.
If you tried to simulate an Event using a mutex, you'd face the problem that as soon as you acquired the lock (when should mean "event signalled"), you're keeping everybody else out until you release that lock. That is not the semantics of signalling an event; it may remain posted, and the "gates" would be open for every thread testing for the event, without acquiring any locks.
Mutex dedicated for interprocess synchronization. This is kernel-mode object.
Events for multithreaded synchronization within one process. This is user-mode object.
Mutex object is very general and to heavy, on the other hand Event object is much more light. In most of situations you must to use user-mode synchronization, because it supplies less CPU cycles.
What happens when you call WaitForSingleObject() on a handle you've created with CreateFile() or _get_osfhandle()?
For reasons not worth explaining I would like to use WaitForSingleObject() to wait on a HANDLE that I've created with _get_osfhandle(fd), where fd comes from a regular call to _open(). Is this possible?
I have tried it in practice, and on some machines it works as expected (the HANDLE is always in the signaled state because you can read more data from it), and on some machines WaitForSingleObject() will block indefinitely if you let it.
The MSDN page for WaitForSingleObject() says that the only supported things that it handles are "change notifications, console input, events, memory resource notifications, mutex, processes, semaphores, threads, and waitable timers."
Additionally, would it be different if I used CreateFile() instead of _get_osfhandle() on a CRT file descriptor?
Don't do it. As you can see, it has undefined behavior.
Even when the behavior is defined, it's defined in such a way as to be relatively not useful unless you don't like writing additional code. It is signaled when any asynchronous I/O operation on that handle completes, which does not generalize to tracking which I/O operation finished.
Why are you trying to wait on a file handle? Clearly the intent matters when you are doing something that isn't even supported well enough to not block indefinitely.
I found the following links. The concensus seems to me, don't do it.
Asynch IO explorer
Waiting on a file handle
When an I/O operation is started on an
asynchronous handle, the handle goes
into a non-signaled state. Therefore,
when used in the context of a
WaitForSingleObject or
WaitForMultipleObjects operation, the
file handle will become signaled when
the I/O operation completes. However,
Microsoft actively discourages this
technique; it does not generalize if
there exists more than one pending I/O
operation; the handle would become
signaled if any I/O operation
completed. Therefore, although this
technique is feasible, it is not
considered best practice.
Egghead Cafe:
Use ReadDirectoryChangesW in
overlapped mode. WaitForSingleObject
can wait on the event in the
OVERLAPPED struct.
You can also use the API
WaitForSingleObject() to wait on a
file change if you use the following
change notification function:
FindFirstChangeNotification()
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/findfirstchangenotification.asp
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/waitforsingleobject.asp
An interesting note on "evilness" of ReadDirectoryChangesW:
http://blogs.msdn.com/ericgu/archive/2005/10/07/478396.aspx