Suppose I have two linux kernel threads, master thread and worker thread. Master uses kthread_run() to create worker. While worker is accepting socket connection and blocking, master calls kthread_stop() to stop worker.
Because worker is blocking on accepting operation and cannot exit, the kthread_stop() inside master will not return.
What should I do to kill worker thread from master in graceful way? Thanks.
You need to specify a timeout for the blocking socket the worker is reading from:
struct timeval tv;
tv.tv_sec = 0; /* 100 ms Timeout */
tv.tv_usec = 100000;
kernel_setsockopt(sock, SOL_SOCKET, SO_RCVTIMEO, (char *)&tv, sizeof(struct timeval));
Whenever the recv call returns you check for kthread_should_stop(). With 100ms timeout there should be almost zero polling overheads.
Related
I have two Win32 console processes a) a server process and b) a client process.
The Server Process:
i) It creates a IO completion port by calling CreateIoCompletionPort(INVALID_HANDLE_VALUE, 0, 0, 1).
ii) Spawn a worker thread and waiting to run the worker thread. The worker thread blocks on GetQueuedCompletionStatus() inside a loop. It simply logs the values which are coming on GetQueuedCompletionStatus().
iii) Create a named pipe (inbound)
iv) Associate the pipe with the IOCP
v) Connect the named pipe and waiting until the overlapped operation completed in the case of getting ERROR_IO_PENDING
vi) The server thread blocks (using GetMessage loop) until it detects any termination message which will be generated on pressing a key to terminate the server process.
vii) On getting a termination message it posts a completion packet to the IOCP to terminate the worker thread.
viii) After worker thread termination, it disconnects the named pipe, close the handles of the named pipe and IOCP.
The Client Process:
i) Open the named pipe by calling CreateFile().
ii) After successful connection, It writes a string to the pipe by calling WriteFile() and waiting for 10 seconds.
Problem: When the client opens the pipe by calling CreateFile() the server worker thread deque a completion packet (where bytes transferred=0) and GetQueuedCompletionStatus() is returning success code. But when the client is writing the buffer to the pipe, nothing is coming to the worker thread and it remains in blocked state though WriteFile() is returning success. I think that I am missing something. Please help.
The problem:
To design an efficient and very fast named-pipes client server framework.
Current state:
I already have battle proven production tested framework. It is fast, however it uses one thread per one pipe connection and if there are many clients the number of threads could fast be to high. I already use smart thread pool (task pool in fact) that can scale with need.
I already use OVERLAPED mode for pipes, but then I block with WaitForSingleObject or WaitForMultipleObjects so that is why I need one thread per connection on the server side
Desired solution:
Client is fine as it is, but on the server side I would like to use one thread only per client request and not per connection. So instead of using one thread for the whole lifecycle of client (connect / disconnect) I would use one thread per task. So only when client requests data and no more.
I saw an example on MSDN that uses array of OVERLAPED structures and then uses WaitForMultipleObjects to wait on them all. I find this a bad design. Two problems I see here. First you have to maintain an array that can grow quite large and deletions will be costly. Second, you have a lot of events, one for each array member.
I also saw completion ports, like CreateIoCompletionPort and GetQueuedCompletionStatus, but I don't see how they are any better.
What I would like is something ReadFileEx and WriteFileEx do, they call a callback routine
when the operation is completed. This is a true async style of programming. But the problem is that ConnectNamedPipe does not support that and furthermore I saw that the thread needs to be in alertable state and you need to call some of the *Ex functions to have that.
So how is such a problem best solved?
Here is how MSDN does it: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365603(v=vs.85).aspx
The problem I see with this approach is that I can't see how you could have 100 clients connected at once if the limit to WaitForMultipleObjects is 64 handles. Sure I can disconnect the pipe after each request, but the idea is to have a permanent client connection just like in TCP server and to track the client through whole life-cycle with each client having unique ID and client specific data.
The ideal pseudo code should be like this:
repeat
// wait for the connection or for one client to send data
Result = ConnectNamedPipe or ReadFile or Disconnect;
case Result of
CONNECTED: CreateNewClient; // we create a new client
DATA: AssignWorkerThread; // here we process client request in a thread
DISCONNECT: CleanupAndDeleteClient // release the client object and data
end;
until Aborted;
This way we have only one listener thread that accepts connect / disconnect / onData events. Thread pool (worker thread) only process the actual request. This way 5 worker threads can serve a lot of clients that are connected.
P.S.
My current code should not be important. I code this in Delphi but its pure WinAPI so the language does not matter.
EDIT:
For now IOCP look like the solution:
I/O completion ports provide an efficient threading model for
processing multiple asynchronous I/O requests on a multiprocessor
system. When a process creates an I/O completion port, the system
creates an associated queue object for requests whose sole purpose is
to service these requests. Processes that handle many concurrent
asynchronous I/O requests can do so more quickly and efficiently by
using I/O completion ports in conjunction with a pre-allocated thread
pool than by creating threads at the time they receive an I/O request.
If server must handle more than 64 events (read/writes) then any solution using WaitForMultipleObjects becomes unfeasible. This is the reason the Microsoft introduced IO completion ports to Windows. It can handle very high number of IO operations using the most appropriate number of threads (usually it's the number of processors/cores).
The problem with IOCP is that it is very difficult to implement right. Hidden issues are spread like mines in the field: [1], [2] (section 3.6). I would recommend using some framework. Little googling suggests something called Indy for Delphi developers. There are maybe others.
At this point I would disregard the requirement for named pipes if that means coding my own IOCP implementation. It's not worth the grief.
I think what you're overlooking is that you only need a few listening named pipe instances at any given time. Once a pipe instance has connected, you can spin that instance off and create a new listening instance to replace it.
With MAXIMUM_WAIT_OBJECTS (or fewer) listening named pipe instances, you can have a single thread dedicated to listening using WaitForMultipleObjectsEx. The same thread can also handle the rest of the I/O using ReadFileEx and WriteFileEx and APCs. The worker threads would queue APCs to the I/O thread in order to initiate I/O, and the I/O thread can use the task pool to return the results (as well as letting the worker threads know about new connections).
The I/O thread main function would look something like this:
create_events();
for (index = 0; index < MAXIMUM_WAIT_OBJECTS; index++) new_pipe_instance(i);
for (;;)
{
if (service_stopping && active_instances == 0) break;
result = WaitForMultipleObjectsEx(MAXIMUM_WAIT_OBJECTS, connect_events,
FALSE, INFINITE, TRUE);
if (result == WAIT_IO_COMPLETION)
{
continue;
}
else if (result >= WAIT_OBJECT_0 &&
result < WAIT_OBJECT_0 + MAXIMUM_WAIT_OBJECTS)
{
index = result - WAIT_OBJECT_0;
ResetEvent(connect_events[index]);
if (GetOverlappedResult(
connect_handles[index], &connect_overlapped[index],
&byte_count, FALSE))
{
err = ERROR_SUCCESS;
}
else
{
err = GetLastError();
}
connect_pipe_completion(index, err);
continue;
}
else
{
fail();
}
}
The only real complication is that when you call ConnectNamedPipe it may return ERROR_PIPE_CONNECTED to indicate that the call succeeded immediately or an error other than ERROR_IO_PENDING if the call failed immediately. In that case you need to reset the event and then handle the connection:
void new_pipe(ULONG_PTR dwParam)
{
DWORD index = dwParam;
connect_handles[index] = CreateNamedPipe(
pipe_name,
PIPE_ACCESS_DUPLEX | FILE_FLAG_OVERLAPPED,
PIPE_TYPE_MESSAGE | PIPE_WAIT | PIPE_ACCEPT_REMOTE_CLIENTS,
MAX_INSTANCES,
512,
512,
0,
NULL);
if (connect_handles[index] == INVALID_HANDLE_VALUE) fail();
ZeroMemory(&connect_overlapped[index], sizeof(OVERLAPPED));
connect_overlapped[index].hEvent = connect_events[index];
if (ConnectNamedPipe(connect_handles[index], &connect_overlapped[index]))
{
err = ERROR_SUCCESS;
}
else
{
err = GetLastError();
if (err == ERROR_SUCCESS) err = ERROR_INVALID_FUNCTION;
if (err == ERROR_PIPE_CONNECTED) err = ERROR_SUCCESS;
}
if (err != ERROR_IO_PENDING)
{
ResetEvent(connect_events[index]);
connect_pipe_completion(index, err);
}
}
The connect_pipe_completion function would create a new task in the task pool to handle the newly connected pipe instance, and then queue an APC to call new_pipe to create a new listening pipe at the same index.
It is possible to reuse existing pipe instances once they are closed but in this situation I don't think it's worth the hassle.
I have the classic IOCP callback that dequeues i/o pending requests, process them, and deallocate them, in this way:
struct MyIoRequest { OVERLAPPED o; /* ... other params ... */ };
bool is_iocp_active = true;
DWORD WINAPI WorkerProc(LPVOID lpParam)
{
ULONG_PTR dwKey;
DWORD dwTrans;
LPOVERLAPPED io_req;
while(is_iocp_active)
{
GetQueuedCompletionStatus((HANDLE)lpParam, &dwTrans, &dwKey, (LPOVERLAPPED*)&io_req, WSA_INFINITE);
// NOTE, i could use GetQueuedCompletionStatusEx() here ^ and set it in the
// alertable state TRUE, so i can wake up the thread with an ACP request from another thread!
printf("dequeued an i/o request\n");
// [ process i/o request ]
...
// [ destroy request ]
destroy_request(io_req);
}
// [ clean up some stuff ]
return 0;
}
Then, in the code I will have somewhere:
MyIoRequest * io_req = allocate_request(...params...);
ReadFile(..., (OVERLAPPED*)io_req);
and this just works perfectly.
Now my question is: What about I want to immediately close the IOCP queue without causing leaks? (e.g. application must exit)
I mean: if i set is_iocp_active to 'false', the next time GetQueuedCompletionStatus() will dequeue a new i/o request, that will be the last i/o request: it will return, causing thread to exit and when a thread exits all of its pending i/o requests are simply canceled by the system, according to MSDN.
But the structures of type 'MyIoRequest' that I have instanced when calling ReadFile() won't be destroyed at all: the system has canceled pending i/o request, but I have to manually destroy those structures I have
created, or I will leak all pending i/o requests when I stop the loop!
So, how I could do this? Am I wrong to stop the IOCP loop with just setting that variable to false? Note that is would happen even if i use APC requests to stop an alertable thread.
The solution that come to my mind is to add every 'MyIoRequest' structures to a queue/list, and then dequeue them when GetQueuedCompletionStatusEx returns, but shouldn't that make some bottleneck, since the enqueue/dequeue process of such MyIoRequest structures must be interlocked? Maybe I've misunderstood how to use the IOCP loop. Can someone bring some light on this topic?
The way I normally shut down an IOCP thread is to post my own 'shut down now please' completion. That way you can cleanly shut down and process all of the pending completions and then shut the threads down.
The way to do this is to call PostQueuedCompletionStatus() with 0 for num bytes, completion key and pOverlapped. This will mean that the completion key is a unique value (you wont have a valid file or socket with a zero handle/completion key).
Step one is to close the sources of completions, so close or abort your socket connections, close files, etc. Once all of those are closed you can't be generating any more completion packets so you then post your special '0' completion; post one for each thread you have servicing your IOCP. Once the thread gets a '0' completion key it exits.
If you are terminating the app, and there's no overriding reason to not do so, (eg. close DB connections, interprocess shared memory issues), call ExitProcess(0).
Failing that, call CancelIO() for all socket handles and process all the cancelled completions as they come in.
Try ExitProcess() first!
When using iocp in a job/task pool to provide fast worker wake ups what is the best way to minimise the overhead of signalling the port - ie not having to do it every queue operation?
void Worker()
{
while(1)
{
for(int spin = 0; spin < 5000; ++spin)
while(queue.Count > 0)
queue.PopFront()();
WaitOnCompletionPort();
}
}
...
queue.PushBack(someWork);
// decide when to signal completion port but avoid doing it every queue operation ?
For example in the above rough code sketch there is a problem between work being queued and the wait being entered if you try and avoid signalling the port every queue operation.
Why don't you use the IOCP as your queue and post your work items directly to it? That way you get a thread safe queue for free and can completely remove the other queue you have?
This question would then go away ;)
In my C++ program I have a class CEvent with trigger and wait member functions based on pthreads (running on Linux). Implementation is quite obvious (i.e. many examples online) if there is one waiting process. However now I need to satisfy the requirement that multiple threads are waiting on the event and should ALL wake up reliably when trigger() is called. As a second condition, only threads that were waiting when trigger() was called should wake up.
My current code:
void CEvent::trigger() {
pthread_mutex_lock(&mutex);
wakeUp = true;
pthread_cond_broadcast(&condition)
pthread_mutex_unlock(&mutex);
wakeUp = false;
}
void CEvent::wait() {
pthread_mutex_lock(&mutex);
while (!wakeUp)
pthread_cond_wait(&condition, &mutex)
pthread_mutex_unlock(&mutex);
}
This seems to almost work, insofar as that all threads waiting wake up before I set wakeUp back to false. However between the broadcast and the reset of wakeUp, other (or the same) threads calling wait() will also wake up right away which is not acceptable. Putting wakeUp = false before the mutext unlocking prevents the threads from waking up.
My questions:
* When does pthread_cond_broadcast return? I.e. is there a guarantee it will only return after all threads have woken up or could it return before?
* Are there any recommended solutions to this problem?
Please disregard my previous bogus answer. There is a race between the time that the trigger thread unlocks the mutex (and thus frees the waiting threads), and then sets the wakeUp value. This means that another (not waiting) thread can come in, grab the mutex, and see a true value in wakeUp and exit without waiting. Another bug is that a thread which was waiting will wake up after wakeUp is reset and immediately resume waiting.
One way to resolve this is to use count - each thread that is waiting will increment the count, then the trigger will wait until that many threads have woken before resuming. You would then have to ensure that non-waiting threads are not allowed to start waiting until this had happened.
// wake up "waiters" count of waiting threads
void CEvent::trigger()
{
pthread_mutex_lock(&mutex);
// wakey wakey
wakeUp = true;
pthread_cond_broadcast(&condition);
// wait for them to awake
while (waiters>0)
pthread_cond_wait(&condition, &mutex);
// stop waking threads up
wakeUp = false;
// let any "other" threads which were ready to start waiting, do so
pthread_cond_broadcast(&condition);
pthread_mutex_unlock(&mutex);
}
// wait for the condition to be notified for us
void CEvent::wait()
{
pthread_mutex_lock(&mutex);
// wait for us to be allowed to start waiting
// we have to wait until any currrently being woken threads have gone
while (wakeUp)
pthread_cond_wait(&condition, &mutex);
// our turn to start waiting
waiters ++;
// waiting
while (!wakeUp)
pthread_cond_wait(&condition, &mutex);
// finished waiting, we were triggered
waiters --;
// let the trigger thread know we're done
pthread_cond_broadcast(&condition);
pthread_mutex_unlock(&mutex);
}