In my kext's stop() function, I call iflt_detach() to detach a registered iff filter. However, it appears that (for whatever reasons), the filter's detach() function may be called outside of the stop() function. In that case, what should I do in the stop function? I can't return KERN_SUCCESS since that would cause the KEXT to get unloaded with obvious side-effects for the delayed call to the detach() function.
The following snippet is from enetlognke.c and shows the stop() function:
kern_return_t com_dts_apple_kext_enetlognke_stop (kmod_info_t * ki, void * d)
{
kern_return_t retval = KERN_FAILURE; // default result, unless we know that we are
// detached from the interface.
if (gFilterRegistered == FALSE)
return KERN_SUCCESS;
if (gUnregisterProc_started == FALSE)
{
// only want to start the detach process once.
iflt_detach(gEnetFilter);
gUnregisterProc_started = TRUE;
}
if (gUnregisterProc_complete)
{
retval = KERN_SUCCESS;
}
else
{
el_printf("enetlognke_stop: incomplete\n");
}
if (retval == KERN_SUCCESS)
{
// Free KEXT resources
}
return retval;
}
gUnregisterProc_complete is set to TRUE from within this module's dispatch() function. So, if that function call is delayed (and gUnregisterProc_complete is FALSE), the stop function would veto the unload by returning KERN_FAILURE.
So, my questions are:
If KERN_FAILURE is returned, will the kernel call the KEXT's stop() function again? If not, what triggers a retry of the KEXT unload and the call to the stop() function?
Is KERN_FAILURE the correct code to return is the filter has not been detached?
Presumably, the detach function will in this case be called on another thread, once there are no threads remaining running your callbacks?
If so, this becomes a fairly straightforward thread synchronisation problem. Set up a flag variable, e.g. has_detached and protect it by a recursive mutex.
In the stop function: Lock the mutex before calling iflt_detach(). If on return, the flag hasn't been set, sleep on the flag's address while suspending the mutex, until the flag is set. Finally, unlock, and return from the stop function.
At the very end of your detach function: lock the mutex, set the flag, send a wakeup to the potentially sleeping thread and unlock. If the unlock call is in the tail position, there is no race condition between executing your detach function's code and unloading said code.
Effectively, this will block the unloading of the kext until your filter has fully detached.
Note: I haven't tried this in this particular case of network filters (I have yet to write a filter kext), but it's generally a pattern I've used a lot in other kexts.
Note 2: I say use a recursive lock to guard against deadlock in case your detach function does get called on the same thread while inside iflt_detach().
Related
int main()
{
tcp::socket socket(iocp);
acceptor.async_accept(socket, yield[ec]);
if (ec)
fail(ec, "accept");
else
boost::asio::spawn(acceptor.get_executor(), std::bind(&do_session, websocket::stream<beast::tcp_stream>(std::move(socket)), std::placeholders::_1));
... iocp run
}
void do_session(websocket::stream<beast::tcp_stream>& ws, net::yield_context yield)
{
while(ws.is_open())
{
ws.async_read(buffer, yield[ec]);
... process the buffer
... execute posted callbacks
}
}
void another_thread()
{
while(isAppNotExit)
{
post_to_specified_coroutine(ws, []() { ... do in courutine same thread });
}
}
I need to post a function in any thread to let the specified coroutine run the function, that is the code part of "execute posted callbacks" above. However, after this task is delivered, the coroutine may be in async_read or async_write state. Is it possible to post an event like data and let the async_read or async_write function return immediately?
I guess the essence of the problem is this: use select on 2 channels: a channel with capacity=1 and a channel with (possibly) infinite capacity.
Implement select with asio asynchronous operation
Write an asio asynchronous operation to wait for multiple (two) things.
(asio asynchronous operation template: c++ - How to wait for a function to return with Boost:::Asio? - Stack Overflow).
state protected by a mutex:
a std::optional<read_result>
a std::vector<functor>
a bool (whether there is an on-going async_read)
a std::optional<completion handler>
your async_wait_for_2_things:
Get the completion handler (a callable, can resume your coroutine) from the completion token (yield[ec]);
Lock the mutex (use guard);
if there is a pending functor from another_thread, take it out, post the completion handler;
else if there is a pending read_result, take it out, post the completion handler;
else if there is a an on-going async_read (the bool is true), store the completion handler (if there is already a completion handler stored, throw "can not happen");
else (no pending functor, no pending read_result, async_read has not been started), store the completion handler (if there is already a completion handler stored, throw "can not happen"), set the bool to true (if the bool is already true, throw "can not happen"), call async_read;
Unlock the mutex;
async_read's callback:
Lock the mutex (use guard);
set the bool to false (if the bool is already false, throw "can not happen");
if there is a completion handler, take it out, post it;
else, store read_result (if there is already a read_result stored, throw "can not happen");
Unlock the mutex;
another_thread's code for posting functor:
Lock the mutex (use guard);
if there is a completion handler, take it out, post it;
else, store functor;
Unlock the mutex;
Implement select using asynchronous event
async_read(use callback overload)'s lambda completion handler: stores result, notifies asynchronous_event;
another_thread: stores functor, notifies asynchronous_event;
do_session: asynchronously waits on asynchronous_event, loads result or functor;
asynchronous_event's data is in a std::pair<std::optional<read_result>, std::vector<functor>> protected by a mutex;
Implement asynchronous event using a timer: c++ - Why does Boost.Asio not support an event-based interface? - Stack Overflow.
This is not applicable because "asynchronous event" is-not-a "asynchronous condition variable", it can not:
release a mutex and block in asynchronous wait atomically
(a possible sequence: do_session release mutex, then functor is posted, then event is notified (cancel_one), then do_session wait on event (timer_.async_wait(yield[ec]);) and blocks forever)
Implement select using asynchronous latch
async_read(use callback overload)'s lambda handler: ①stores result and resets asynchronous_latch_producer, ②notifies asynchronous_latch_consumer, waits on asynchronous_latch_producer(, ⑥wake up);
another_thread: ①stores functor and resets asynchronous_latch_producer, ②notifies asynchronous_latch_consumer, waits on asynchronous_latch_producer(, ⑥wake up);
do_session: waits on asynchronous_latch_consumer(, ③wake up), ④loads result or functor and resets asynchronous_latch_consumer, ⑤notifies asynchronous_latch_producer;
asynchronous_latch_consumer and asynchronous_latch_producer's data is in a std::pair<std::optional<read_result>, std::vector<functor>>;
Implement asynchronous latch using a timer: c++ - Cancelling boost asio deadline timer safely - Stack Overflow. Modify that asynchronous event implementation to get asynchronous latch: in constructor and reset, .expires_at(Timer::clock_type::time_point::max()); in notify_all_one_shot, .expires_at(Timer::clock_type::time_point::min()).
This is not applicable because one of the producer might block forever.
I am trying to understand how wait_event is implemented in linux kernel. There is a code example in ldd3 where the internal implementation is explained using prepare_to_wait (http://www.makelinux.net/ldd3/chp-6-sect-2).
static int scull_getwritespace(struct scull_pipe *dev, struct file *filp)
{
while (spacefree(dev) == 0) {
DEFINE_WAIT(wait);
up(&dev->sem);
if (filp->f_flags & O_NONBLOCK)
return -EAGAIN;
PDEBUG("\"%s\" writing: going to sleep\n",current->comm);
prepare_to_wait(&dev->outq, &wait, TASK_INTERRUPTIBLE);
if (spacefree(dev) == 0) // Why is this check necessary ??
schedule( );
finish_wait(&dev->outq, &wait);
if (signal_pending(current))
return -ERESTARTSYS; /* signal: tell the fs layer to handle it */
if (down_interruptible(&dev->sem))
return -ERESTARTSYS;
}
return 0;
}
In the book, it is explained as below.
Then comes the obligatory check on the buffer; we must handle the case
in which space becomes available in the buffer after we have entered
the while loop (and dropped the semaphore) but before we put ourselves
onto the wait queue. Without that check, if the reader processes were
able to completely empty the buffer in that time, we could miss the
only wakeup we would ever get and sleep forever. Having satisfied
ourselves that we must sleep, we can call schedule.
I am not able to understand this piece of explanation. How we would go to a indefinite sleep if the if (spacefree(dev) == 0) is not done before calling schedule() ?
if this obligatory check is not present, wakeup() still resets the process state to TASK_RUNNING and schedule returns as explained in the next paragraph.
It is worth looking again at this case: what happens if the wakeup
happens between the test in the if statement and the call to schedule?
In that case, all is well. The wakeup resets the process state to
TASK_RUNNING and schedule returns—although not necessarily right away.
As long as the test happens after the process has put itself on the
wait queue and changed its state, things will work.
The important thing is that the (last) check is done after prepare_to_wait() was called.
prepare_to_wait() puts a pointer to the current process into the wait queue. If the wakeup happens before the prepare_to_wait() call, the wakeup would not be able to affect the current process.
Using the Windows Mutex functions to make an application one instance I'm wondering how to tell if the Mutex object, if it exists, is 'owned' or not so I can ignore it being a valid object should the previous instance have crashed?
Your main goal is to have a single instance of the application.
You could create a mutex without acquiring it, set bInitialOwner to FALSE, so you can use as a label.
On the start up, check if the mutex exits, if so, cleanup, e.g. notify the existing process, and exit.
If not, create one without acquiring it.
for example:
HANDLE Mutex;
DWORD Error;
Mutex = CreateMutex(NULL, FALSE, TEXT("UniqueMutexName"));
Error = GetLastError();
if(Mutex != NULL && Error == ERROR_ALREADY_EXISTS)
{
/* another instance running */
CloseHandle(Mutex);
ExitProcess(0);
}
else if(Mutex == NULL)
{
/* different error */
SetLastError(Error);
}
...
CloseHandle(Mutex);
If you want to check if the mutex is owned, you can call WaitForSingleObject with zero timeout:
switch(WaitForSingleObject(Mutex, 0))
{
case WAIT_ABANDONED:
/* similar to the bellow, but be careful with this one, if
* there's some protected shared data it may left corrupted */
case WAIT_OBJECT_0:
/* was not acquired, you just acquired it */
ReleaseMutex(Mutex);
break;
case WAIT_TIMEOUT:
/* already owned */
break;
default:
/* some error */
}
If the process was terminated or crushed without calling CloseHandle, the system will close the handle automatically, CreateMutex:
Use the CloseHandle function to close the handle. The system closes
the handle automatically when the process terminates. The mutex object
is destroyed when its last handle has been closed.
I want to implement a scheduler class, which any object can use to schedule timeouts and cancel then if necessary. When a timeout expires, this information will be sent to the timeout setter/owner at that time asynchronously.
So, for this purpose, I have 2 fundamental classes WindowsTimeout and WindowsScheduler.
class WindowsTimeout
{
bool mCancelled;
int mTimerID; // Windows handle to identify the actual timer set.
ITimeoutReceiver* mSetter;
int cancel()
{
mCancelled = true;
if ( timeKillEvent(mTimerID) == SUCCESS) // Line under question # 1
{
delete this; // Timeout instance is self-destroyed.
return 0; // ok. OS Timer resource given back.
}
return 1; // fail. OS Timer resource not given back.
}
WindowsTimeout(ITimeoutReceiver* setter, int timerID)
{
mSetter = setter;
mTimerID = timerID;
}
};
class WindowsScheduler
{
static void CALLBACK timerFunction(UINT uID,UINT uMsg,DWORD dwUser,DWORD dw1,DWORD dw2)
{
WindowsTimeout* timeout = (WindowsTimeout*) uMsg;
if (timeout->mCancelled)
delete timeout;
else
timeout->mDestination->GEN(evTimeout(timeout));
}
WindowsTimeout* schedule(ITimeoutReceiver* setter, TimeUnit t)
{
int timerID = timeSetEvent(...);
if (timerID == SUCCESS)
{
return WindowsTimeout(setter, timerID);
}
return 0;
}
};
My questions are:
Q.1. When a WindowsScheduler::timerFunction() call is made, this call is performed in which context ? It is simply a callback function and I think, it is performed by the OS context, right ? If it is so, does this calling pre-empt any other tasks already running ? I mean do callbacks have higher priority than any other user-task ?
Q.2. When a timeout setter wants to cancel its timeout, it calls WindowsTimeout::cancel().
However, there is always a possibility that timerFunction static call to be callbacked by OS, pre-empting the cancel operation, for example, just after mCancelled = true statement. In such a case, the timeout instance will be deleted by the callback function.
When the pre-empted cancel() function comes again, after the callback function completes execution, will try to access an attribute of the deleted instance (mTimerID), as you can see on the line : "Line under question # 1" in the code.
How can I avoid such a case ?
Please note that, this question is an improved version of the previos one of my own here:
Windows multimedia timer with callback argument
Q1 - I believe it gets called within a thread allocated by the timer API. I'm not sure, but I wouldn't be surprised if the thread ran at a very high priority. (In Windows, that doesn't necessarily mean it will completely preempt other threads, it just means it will get more cycles than other threads).
Q2 - I started to sketch out a solution for this, but then realized it was a bit harder than I thought. Personally, I would maintain a hash table that maps timerIDs to your WindowsTimeout object instances. The hash table could be a simple std::map instance that's guarded by a critical section. When the timer callback occurs, it enters the critical section and tries to obtain the WindowsTimer instance pointer, and then flags the WindowsTimer instance as having been executed, exits the critical section, and then actually executes the callback. In the event that the hash table doesn't contain the WindowsTimer instance, it means the caller has already removed it. Be very careful here.
One subtle bug in your own code above:
WindowsTimeout* schedule(ITimeoutReceiver* setter, TimeUnit t)
{
int timerID = timeSetEvent(...);
if (timerID == SUCCESS)
{
return WindowsTimeout(setter, timerID);
}
return 0;
}
};
In your schedule method, it's entirely possible that the callback scheduled by timeSetEvent will return BEFORE you can create an instance of WindowsTimeout.
I know that I can use condition variable to synchronize work between the threads, but is there any class like this (condition variable) to synchronize work between the processes, thanks in advance
Use a pair of named Semaphore objects, one to signal and one as a lock. Named sync objects on Windows are automatically inter-process, which takes care of that part of the job for you.
A class like this would do the trick.
class InterprocessCondVar {
private:
HANDLE mSem; // Used to signal waiters
HANDLE mLock; // Semaphore used as inter-process lock
int mWaiters; // # current waiters
protected:
public:
InterprocessCondVar(std::string name)
: mWaiters(0), mLock(NULL), mSem(NULL)
{
// NOTE: You'll need a real "security attributes" pointer
// for child processes to see the semaphore!
// "CreateSemaphore" will do nothing but give you the handle if
// the semaphore already exists.
mSem = CreateSemaphore( NULL, 0, std::numeric_limits<LONG>::max(), name.c_str());
std::string lockName = name + "_Lock";
mLock = CreateSemaphore( NULL, 0, 1, lockName.c_str());
if(!mSem || !mLock) {
throw std::runtime_exception("Semaphore create failed");
}
}
virtual ~InterprocessCondVar() {
CloseHandle( mSem);
CloseHandle( mLock);
}
bool Signal();
bool Broadcast();
bool Wait(unsigned int waitTimeMs = INFINITE);
}
A genuine condition variable offers 3 calls:
1) "Signal()": Wake up ONE waiting thread
bool InterprocessCondVar::Signal() {
WaitForSingleObject( mLock, INFINITE); // Lock
mWaiters--; // Lower wait count
bool result = ReleaseSemaphore( mSem, 1, NULL); // Signal 1 waiter
ReleaseSemaphore( mLock, 1, NULL); // Unlock
return result;
}
2) "Broadcast()": Wake up ALL threads
bool InterprocessCondVar::Broadcast() {
WaitForSingleObject( mLock, INFINITE); // Lock
bool result = ReleaseSemaphore( mSem, nWaiters, NULL); // Signal all
mWaiters = 0; // All waiters clear;
ReleaseSemaphore( mLock, 1, NULL); // Unlock
return result;
}
3) "Wait()": Wait for the signal
bool InterprocessCondVar::Wait(unsigned int waitTimeMs) {
WaitForSingleObject( mLock, INFINITE); // Lock
mWaiters++; // Add to wait count
ReleaseSemaphore( mLock, 1, NULL); // Unlock
// This must be outside the lock
return (WaitForSingleObject( mSem, waitTimeMs) == WAIT_OBJECT_0);
}
This should ensure that Broadcast() ONLY wakes up threads & processes that are already waiting, not all future ones too. This is also a VERY heavyweight object. For CondVars that don't need to exist across processes I would create a different class w/ the same API, and use unnamed objects.
You could use named semaphore or named mutex. You could also share memory between processes by shared memory.
For a project I'm working on I needed a condition variable and mutex implementation which can handle dead processes and won't cause other processes to end up in a deadlock in such a case. I implemented the mutex with the native named mutexes provided by the WIN32 api because they can indicate whether a dead process owns the lock by returning WAIT_ABANDONED. The next issue was that I also needed a condition variable I could use across processes together with these mutexes. I started of with the suggestion from user3726672 but soon discovered that there are several issues in which the state of the counter variable and the state of the semaphore ends up being invalid.
After doing some research, I found a paper by Microsoft Research which explains exactly this scenario: Implementing Condition Variables with Semaphores . It uses a separate semaphore for every single thread to solve the mentioned issues.
My final implementation uses a portion of shared memory in which I store a ringbuffer of thread-ids (the id's of the waiting threads). The processes then create their own handle for every named semaphore/thread-id which they have not encountered yet and cache it. The signal/broadcast/wait functions are then quite straight forward and follow the idea of the proposed solution in the paper. Just remember to remove your thread-id from the ringbuffer if your wait operation fails or results in a timeout.
For the Win32 implementation I recommend reading the following documents:
Semaphore Objects and Using Mutex Objects as those describe the functions you'll need for the implementation.
Alternatives: boost::interprocess has some robust mutex emulation support but it is based on spin locks and caused a very high cpu load on our embedded system which was the final reason why we were looking into our own implementation.
#user3726672: Could you update your post to point to this post or to the referenced paper?
Best Regards,
Michael
Update:
I also had a look at an implementation for linux/posix. Turns out pthread already provides everything you'll need. Just put pthread_cond_t and pthread_mutex_t in some shared memory to share it with the other process and initialize both with PTHREAD_PROCESS_SHARED. Also set PTHREAD_MUTEX_ROBUST on the mutex.
Yes. You can use a (named) Mutex for that. Use CreateMutex to create one. You then wait for it (with functions like WaitForSingleObject), and release it when you're done with ReleaseMutex.
For reference, Boost.Interprocess (documentation for version 1.59) has condition variables and much more. Please note, however, that as of this writing, that "Win32 synchronization is too basic".