I have 5 threads (on a multicore system) that simultaneously wait for:
A mutex M to be acquired
An event E to be signaled
I'm using WaitForMultipleObjectsEx(..., TRUE, INFINITE, TRUE) since the threads use APCs.
Note that the APCs do not use/touch/alter/observe the mutex or the event in any way.
I'm using Visual Studio along with Process Hacker to debug, and I'm witnessing a strange phenomenon:
Frequently, the thread that is supposed to run (call it thread A) is still waiting on the two objects.
I say "supposed to run" because:
I have verified many times (including via Process Hacker) that the E is indeed in a signaled state
No other thread has acquired ownership of M. In fact, 3 other threads are also waiting to acquire M.
Nevertheless, according to Process Hacker (which uses NtQueryMutant to retrieve the count), during such scenarios, M has an absurdly low "Count" value, such as -618.
Of course, I'm not forgetting to call ReleaseMutex (yes, I'm checking the return value too); indeed, once someone later acquires the mutex, the Count increases back up.
So this begs the question: if 3 of the 4 other threads are sleeping, what is the last thread (B) doing?
The answer is that B is executing an APC inside the WaitForMultipleObjectsEx call.
No matter how far I "step" through B in Visual Studio (even all the way back up RtlDispatchAPC), none of the other threads wake up.
It is only after that APC has finished executing that any thread wakes up!
I find this quite bizarre, WaitForMultipleObjectEx's documentation says:
When bWaitAll is TRUE, the function's wait operation is completed only when the states of all objects have been set to signaled. The function does not modify the states of the specified objects until the states of all objects have been set to signaled. For example, a mutex can be signaled, but the thread does not get ownership until the states of the other objects are also set to signaled. In the meantime, some other thread may get ownership of the mutex, thereby setting its state to nonsignaled.
Is the documentation wrong, or is this a mistake on my part?
Why does WaitForMultipleObjectsEx acquire the mutex during the execution of an APC, even though it knows it will never acquire the mutex anyway (because executing an APC implies it will return WAIT_IO_COMPLETE, never acquiring the mutex)?
I cannot comment so I use answer...
You don't show any code but shouldn't the APC trigger the WaitForMultipleObjectsEx return with WAIT_IO_COMPLETION code in the thread that is running??
Related
I have hand-made thread pool. Threads read from completion port and do some other stuff. One particular thread has to be ended. How to interrupt it's waiting if it hangs on GetQueuedCompletionStatus() or GetQueuedCompletionStatusEx()?
Finite timeout (100-1000 ms) and exiting variable are far from elegant, cause delays and left as last resort.
CancelIo(completionPortHandle) within APC in target thread causes ERROR_INVALID_HANDLE.
CancelSynchronousIo(completionPortHandle) causes ERROR_NOT_FOUND.
PostQueuedCompletionStatus() with termination packet doesn't allow to choose thread.
Rough TerminateThread() with mutex should work. (I haven't tested it.) But is it ideologically good?
I tried to wait on special event and completion port. WaitForMultipleObjects() returned immediately as if completion port was signalled. GetQueuedCompletionStatus() shows didn't return anything.
I read Overlapped I/O: How to wake a thread on a completion port event or a normal event? and googled a lot.
Probably, the problem itself – ending thread's work – is sign of bad design and all my threads should be equal and compounded into normal thread pool. In this case, PostQueuedCompletionStatus() approach should work. (Although I have doubts that this approach is beautiful and laconic especially if threads use GetQueuedCompletionStatusEx() to get multiple packets at once.)
If you just want to reduce the size of the thread pool it doesn't matter which thread exits.
However if for some reason you need to signal to an particular thread that it needs to exit, rather than allowing any thread to exit, you can use this method.
If you use GetQueuedCompletionStatusEx you can do an alertable wait, by passing TRUE for fAlertable. You can then use QueueUserAPC to queue an APC to the thread you want to quit.
https://msdn.microsoft.com/en-us/library/windows/desktop/ms684954(v=vs.85).aspx
If the thread is busy then you will still have to wait for the current work item to be completed.
Certainly don't call TerminateThread.
Unfortunately, I/O completion port handles are always in a signaled state and as such cannot really be used in WaitFor* functions.
GetQueuedCompletionStatus[Ex] is the only way to block on the completion port. With an empty queue, the function will return only if the thread becomes alerted. As mentioned by #Ben, the QueueUserAPC will make the the thread alerted and cause GetQueuedCompletionStatus to return.
However, QueueUserAPC allocates memory and thus can fail in low-memory conditions or when memory quotas are in effect. The same holds for PostQueuedCompletionStatus. As such, using any of these functions on an exit path is not a good idea.
Unfortunately, the only robust way seems to be calling the undocumented NtAlertThread exported by ntdll.dll.
extern "C" NTSTATUS __stdcall NtAlertThread(HANDLE hThread);
Link with ntdll.lib. This function will put the target thread into an alerted state without queuing anything.
may be I misunderstood something but...
When I call pthread_mutex_lock() and then call pthread_mutex_lock() out of the same thread again without calling pthread_mutex_unlock(), the second call of pthread_mutex_lock() will block.
But: when I call EnterCriticalSection() and call EnterCriticalSection() out of the same thread again without calling LeaveCriticalSection(), the second call of EnterCriticalSection() will NOT block since it is called out of the same thread (what is a very weird behaviour for me).
So my question is there a WinAPI function available that behaves like pthread_mutex_lock() and locks independent from the thread context?
I'm aware of libpthread for Windows but I prefer to have a WinAPI function here.
You could use a Semaphore with the maximum count set to one.
See Semaphore Objects
When you successfully acquire the semaphore, its count is decremented: going to zero in our case.
No other thread can acquire it, including the current one.
pthread_mutex_lock documentation:
If the mutex type is PTHREAD_MUTEX_RECURSIVE, then the mutex maintains the concept of a lock count. When a thread successfully acquires a mutex for the first time, the lock count is set to one. Every time a thread relocks this mutex, the lock count is incremented by one. Each time the thread unlocks the mutex, the lock count is decremented by one. When the lock count reaches zero, the mutex becomes available for other threads to acquire. If a thread attempts to unlock a mutex that it has not locked or a mutex which is unlocked, an error will be returned.
MSDN ReleaseMutex states:
A thread can specify a mutex that it already owns in a call to one of the wait functions without blocking its execution. This prevents a thread from deadlocking itself while waiting for a mutex that it already owns. However, to release its ownership, the thread must call ReleaseMutex one time for each time that it obtained ownership (either through CreateMutex or a wait function).
The wait functions are the equivalent to pthread_mutex_lock.
See Mutex Objects (Windows) to get more details about this API.
And this stackoverflow entry to see what the CRITICAL_SECTION object contains. This will disclose
that the CRITICAL_SECTION object holds - among others - a value LockCount to allow recursive use. See the EnterCriticalSection function to learn about this feature.
http://msdn.microsoft.com/en-us/library/windows/desktop/ms686289%28v=vs.85%29.aspx
According to msdn, in the remarks sections, it states:
"If the thread that set the timer terminates and there is an associated completion routine, the timer is canceled. However, the state of the timer remains unchanged. If there is no completion routine, then terminating the thread has no effect on the timer."
Then further down, it states:
"If the thread that called SetWaitableTimer exits, the timer is canceled. This stops the timer before it can be set to the signaled state and cancels outstanding APCs; it does not change the signaled state of the timer."
Hence my question,
if I have one thread calling SetWaitableTimer without an associated completion routine and another thread calling WaitOnMultipleObjects(passing in the timer object handle) and the thread that calls SetWaitiableTmer exits shortly thereafter, would the timer object be cancelled or would it still become signaled when the period expires?
To give more information directly from the implementation of waitable timers: if you use a CompletionRoutine, the timer is placed on a linked list chained off the thread which called SetWaitableTimer. When the thread is terminated, the kernel walks the dying thread's linked list and cancels are timers which are still queued.
If you're not using a completion routine, the timer is never added to any thread's linked list and thus isn't cancelled when any particular thread dies.
The documentation is somewhat unclear. I think the best you can do is test it yourself. I believe however that the timer cancels automatically only if the I/O completion routine is used.
I can give some "theoretical" background about windows APCs, to justify my (educated) guess.
APC = "asynchronous procedure call". In windows every user-mode thread is equipped with a so-called APC queue, a system-managed queue of procedures that must be called on this thread. A thread may enter a so-called "alertable wait" state (on purpose), during which it may execute one or more of the procedures in this queue. You may either put the procedure call in the APC queue manually, or issue an I/O, which on completion will "put" the procedure call there.
In simple words the scenario is the following: you issue several I/Os, and then you wait for either of them to complete (or fail), and, perhaps, some other events. You then call one of the alertable-waiting functions: SleepEx, WaitForMultipleObjectsEx or similar.
Important note: this mechanism is designed to support a single-threaded concurrency. That is, the same thread issues several I/Os, waits for something to happen, and responds appropriately. All the APC routines are guaranteed to be called in the same thread. Hence - if this thread exits - there's no way to call them. Hence - all the outstanding I/Os are also cancelled.
There are several Windows API functions that deal with asynchronous I/O, whereas they allow a choice of several completion mechanisms (such as ReadFileEx): APC, setting an event, or putting a completion in the I/O completion port. If those functions are used with APC - they automatically cancel the I/O if the issuing thread exits.
Hence, I guess that waitable timer auto-cancels only if used with APC.
Is there a way to force a context switch in C++ to a specific thread, assuming I have the thread handle or thread ID?
No, you won't be able to force operating system to run the thread you want. You can use yield to force a context switch though...
yield in Win32 API is function SwitchToThread. If there is no other thread available for running, then a ZERO value will be returned and current thread will keep running anyway.
You can only encourage the Windows thread scheduler to pick a certain thread, you can't force it. You do so first by making the thread block on a synchronization object and signaling it. Secondary by bumping up its priority.
Explicit context switching is supported, you'll have to use fibers. Review SwitchToFiber(). A fiber is not a thread by a long shot, it is similar to a co-routine of old. Fibers' heyday has come and gone, they are not competitive with threads anymore. They have very crappy cpu cache locality and cannot take advantage of multiple cores.
The only way to force a particular thread to run is by using process/thread affinity, but I can't imagine ever having a problem for which this was a reasonable solution.
The only way to force a context switch is to force a thread onto a different processor using affinity.
In other words, what you are trying to do isn't really viable.
Calling SwitchToThread() will result in a context switch if there is another thread ready to run that are eligible to run on this processor. The documentation states it as follows:
If calling the SwitchToThread function
causes the operating system to switch
execution to another thread, the
return value is nonzero.
If there are no other threads ready to
execute, the operating system does not
switch execution to another thread,
and the return value is zero.
You can temporarily bump the priority of the other thread, while looping with Sleep(0) calls: this passes control to other threads. Suppose that the other thread has increased a lock variable and you need to wait until it becomes zero again:
// Wait until other thread releases lock
SetThreadPriority(otherThread, THREAD_PRIORITY_HIGHER);
while (InterlockedRead(&lock) != 0)
Sleep(0);
SetThreadPriority(otherThread, THREAD_PRIORITY_NORMAL);
I would check out the book Concurrent Programming for Windows. The scheduler seems to do a few things worth noting.
Sleep(0) only yields to higher priority threads (or possibly others at the same priority). This means you cannot fix priority inversion situations with just a Sleep(0), where other lower priority threads need to run. You must use SwitchToThread, Sleep a non-zero duration, or fully block on some kernel HANDLE.
You can create two synchronization objects (such as two events) and use the API SignalObjectAndWait.
If the hObjectToWaitOn is non-signaled and your other thread is waiting on the hObjectToSignal, the OS can theoretically perform quick context switch inside this API, before end of time slice.
And if you want the current thread to automatically resume, simply inform a small value (such as 50 or 100) on the dwMilliseconds.
i'm not sure about something.
when i use critical_section/mutex/semaphor in c++ for example , how does the busy_wait problem being prevented ?
what i mean is when a thread reaches a critical section and the critical section is occupied by other thread, what prevents the thread from wasting cycle time and wait for nothing ?
for example,
should i call TryEnterCriticalSection and check if the thread obtained ownership and otherwise call sleep(0) ?
i'm a bit perplexed
thanks
This is Windows specific, but Linux will be similar.
Windows has the concept of a ready queue of threads. These are threads that are ready to run, and will be run at some point on an available processor. Which threads are selected to run immediately is a bit complicated - threads can have different priorities, their priorities can be temporarily boosted, etc.
When a thread waits on a synchronization primitive like a CRITICAL_SECTION or mutex, it is not placed on the ready queue - Windows will not even attempt to run the thread and will run other threads if possible. At some point the thread will be moved back to the ready queue, for instance when the thread owning the CS or mutex releases it.
The thread is not going to be taking any system resources, because it will be marked as "waiting". As soon as the thread occupying the critical region finishes, it will send out a signal that will move the waiting thread to the ready queue.
These control structures stop the thread that can't enter from doing a busy wait by allowing it to sleep until an interrupt is generated by the thread that is in the critical section finishing execution. Because the thread is asleep it is not using processor cycles, so no busy_wait.