HWM does not seem to work in clrzmq 2.2.5.
Here's my code
private static ulong hwm = 50;
static void testMQ()
{
var _Context = new Context(1);
var pubSock = _Context.Socket(SocketType.PUB);
pubSock.HWM = hwm;
pubSock.Bind("tcp://*:9999");
new Thread(testSub).Start();
Thread.Sleep(1000); // client connect
int i = 0;
while (true)
{
pubSock.Send(i.ToString(), Encoding.ASCII);
Debug.WriteLine(pubSock.Backlog + "/" + i++);
}
}
static void testSub()
{
var _ZmqCtx = new Context(1);
var subSock = _ZmqCtx.Socket(SocketType.SUB);
subSock.HWM = 500;
subSock.Identity = new ASCIIEncoding().GetBytes("bla");
subSock.Connect("tcp://127.0.0.1:9999");
Debug.WriteLine("connected");
subSock.Subscribe("", Encoding.ASCII);
while (true)
{
Debug.WriteLine("r:" + subSock.Recv(Encoding.ASCII));
Thread.Sleep(10);
}
}
Output:
'quickies.vshost.exe' (Managed (v4.0.30319)):
Loaded 'B:\sdev\MSenseWS\GoogleImporter\bin\Debug\clrzmq.dll', Symbols loaded.
connected
r:0
100/0
100/1
100/2
[...]
100/13
r:1
100/14
[...]
100/2988
100/2989
100/2990
100/2991
100/2992
100/2993
100/2994
100/2995
100/2996
r:179
100/2997
100/2998
Expected behavior: pubSock.Send blocks after 500 messages are queued.
Experienced behavior: pubSock.Sends does not block and sends forever until out of memory exception vom native code (clrzmq.dll) is thrown.
Also: Why is backlog always 100?
Thanks for your insights,
Armin
Edit: push/poll sockets achieve the same result
#
#
Resolution:
- The error was on my side, as i was expecting that the HWM is the number of outstanding messages that the clinet(s) have not commited (received). While in fact HWM is the number of messages that are buffered and queued for sending over the network.
In my case i had a client that can not process messages fast enough and so buffer space was allocated until out of memory.
To solve this problem i found out that setting HWM and SWAP on the client socket solves my problem, as messages are queued to a large swap file by zmq and are successively precessed by the application.
Ah, so I'm guessing you have the subscriber thread sleep, but that doesn't mean the underlying ZMQ socket threads also sleep. Therefore the subscriber will continue to take messages off the publisher queue. In other words, using Thread.Sleep() is probably not a good enough way to simulate limited network connectivity or other issues you expect to cause running into the HWM.
Related
I need to pass data between CoreAudio's realtime thread and the UI thread (one way, RT->UI). I know I can't use any Cocoa/Objective C methods like performSelectorOnMainThread or NSNotification and I can't use anything that will allocate memory as this will potentially block the RT thread.
What is the correct method for communicating between threads? Can I use GCD message queues or is there a more basic system to use?
Edit:
Thinking about this a bit more, I suppose I could use a lock free ring buffer, which the RT thread puts a message into, and the UI thread checks for messages to pull out. Is this the best way and if so is there a system already to do this in CoreAudio or available elsewhere or do I need to code it up myself?
It turns out this was a lot simpler than I expected and the solution I came up with was just to use the Portaudio ring buffer. I needed to add pa_ringbuffer.[ch] and pa_memorybarrier.h to my project and then define a MessageData structure to store in the ring buffer.
typedef struct MessageData {
MessageType type;
union {
struct {
NSUInteger position;
} position;
} data;
} MessageData;
Then I allocated some space to store 32 messages and created the ring buffer.
_playbackData->RTToMainBuffer = malloc(sizeof(MessageData) * 32);
PaUtil_InitializeRingBuffer(&_playbackData->RTToMainRB, sizeof(MessageData),
32, _playbackData->RTToMainBuffer);
Finally I started an NSTimer for every 20ms to pull data from the ring buffer
while (PaUtil_GetRingBufferReadAvailable(&_playbackData->RTToMainRB)) {
MessageData *dataPtr1, *dataPtr2;
ring_buffer_size_t sizePtr1, sizePtr2;
// Should we read more than one at a time?
if (PaUtil_GetRingBufferReadRegions(&_playbackData->RTToMainRB, 1,
(void *)&dataPtr1, &sizePtr1,
(void *)&dataPtr2, &sizePtr2) != 1) {
continue;
}
// Parse message
switch (dataPtr1->type) {
case MessageTypeEOS:
break;
case MessageTypePosition:
break;
default:
break;
}
PaUtil_AdvanceRingBufferReadIndex(&_playbackData->RTToMainRB, 1);
}
Then in the realtime thread, pushing a message to the ringbuffer was simply
MessageData *dataPtr1, *dataPtr2;
ring_buffer_size_t sizePtr1, sizePtr2;
if (PaUtil_GetRingBufferWriteRegions(&data->RTToMainRB, 1,
(void *)&dataPtr1, &sizePtr1,
(void *)&dataPtr2, &sizePtr2)) {
dataPtr1->type = MessageTypePosition;
dataPtr1->data.position.position = currentPosition;
PaUtil_AdvanceRingBufferWriteIndex(&data->RTToMainRB, 1);
}
A ringbuffer is a good solution. Two if you need to communicate both ways ie. inbox/outbox message passing.
This is a good implementation for iOS/Mac if you don't want to use Portaudio.
https://github.com/michaeltyson/TPCircularBuffer
I need to run unsafe native code on a sandbox process and I need to reduce bottleneck of process switch. Both processes (controller and sandbox) shares two auto-reset events and a coherent view of a mapped file (shared memory) that is used for communication.
To make this article smaller, I removed initializations from sample code, but the events are created by the controller, duplicated using DuplicateHandle, and then sent to sandbox process prior to work.
Controller source:
void inSandbox(HANDLE hNewRequest, HANDLE hAnswer, volatile int *shared) {
int before = *shared;
for (int i = 0; i < 100000; ++i) {
// Notify sandbox of a new request and wait for answer.
SignalObjectAndWait(hNewRequest, hAnswer, INFINITE, FALSE);
}
assert(*shared == before + 100000);
}
void inProcess(volatile int *shared) {
int before = *shared;
for (int i = 0; i < 100000; ++i) {
newRequest(shared);
}
assert(*shared == before + 100000);
}
void newRequest(volatile int *shared) {
// In this test, the request only increments an int.
(*shared)++;
}
Sandbox source:
void sandboxLoop(HANDLE hNewRequest, HANDLE hAnswer, volatile int *shared) {
// Wait for the first request from controller.
assert(WaitForSingleObject(hNewRequest, INFINITE) == WAIT_OBJECT_0);
for(;;) {
// Perform request.
newRequest(shared);
// Notify controller and wait for next request.
SignalObjectAndWait(hAnswer, hNewRequest, INFINITE, FALSE);
}
}
void newRequest(volatile int *shared) {
// In this test, the request only increments an int.
(*shared)++;
}
Measurements:
inSandbox() - 550ms, ~350k context switches, 42% CPU (25% kernel, 17% user).
inProcess() - 20ms, ~2k context switches, 55% CPU (2% kernel, 53% user).
The machine is Windows 7 Pro, Core 2 Duo P9700 with 8gb of memory.
An interesting fact is that sandbox solution uses 42% of CPU vs 55% of in-process solution. Another noteworthy fact is that sandbox solution contains 350k context switches, which is much more than the 200k context switches that we can infer from source code.
I need to know if there's a way to reduce the overhead of transfer control to another process. I already tried to use pipes instead of events, and it was much worse. I also tried to use no event at all, by making the sandbox call SuspendThread(GetCurrentThread()) and making the controller call ResumeThread(hSandboxThread) on every request, but the performance was similar to using events.
If you have a solution that uses assembly (like performing a manual context switch) or Windows Driver Kit, please let me know as well. I don't mind having to install a driver to make this faster.
I heard that Google Native Client does something similar, but I only found this documentation. If you have more information, please let me know.
The first thing to try is raising the priority of the waiting thread. This should reduce the number of extraneous context switches.
Alternatively, since you're on a 2-core system, using spinlocks instead of events would make your code much much faster, at the cost of system performance and power consumption:
void inSandbox(volatile int *lock, volatile int *shared)
{
int i, before = *shared;
for (i = 0; i < 100000; ++i) {
*lock = 1;
while (*lock != 0) { }
}
assert(*shared == before + 100000);
}
void newRequest(volatile int *shared) {
// In this test, the request only increments an int.
(*shared)++;
}
void sandboxLoop(volatile int *lock, volatile int * shared)
{
for(;;) {
while (*lock != 1) { }
newRequest(shared);
*lock = 0;
}
}
In this scenario, you should probably set thread affinity masks and/or lower the priority of the spinning thread so that it doesn't compete with the busy thread for CPU time.
Ideally, you'd use a hybrid approach. When one side is going to be busy for a while, let the other side wait on an event so that other processes can get some CPU time. You could trigger the event a little ahead of time (using the spinlock to retain synchronization) so that the other thread will be ready when you are.
I have an asynchronous socket and call to connect() + GetLastError() which returns WSA_WOULD_BLOCK, as expected. So I start "receiving/reading" thread and subscribe Event to FD_READ and FD_CLOSE.
The story is: connect will sequentially fail, since Server is not up and running. My understanding that my receiving thread should get FD_CLOSE soon and I need to follow-up with cleaning.
It does not happen. How soon should I receive FD_CLOSE? Is it proper approach? Is there any other way to understand that connect() failed? Shoul I ever receive FD_CLOSE if socket isn't connected?
I do start my receiving thread and subscribe event after successful call to DoConnect() and I am afraid that racing condition prevents me from getting FD_CLOSE.
Here is some code:
int RecvSocketThread::WaitForData()
{
int retVal = 0
while (!retVal)
{
// sockets to pool can be added on other threads.
// please validate that all of them in the pool are connected
// before doing any reading on them
retVal = DoWaitForData();
}
}
int RecvSocketThread::DoWaitForData()
{
// before waiting for incoming data, check if all sockets are connected
WaitForPendingConnection_DoForAllSocketsInThePool();
// other routine to read (FD_READ) or react to FD_CLOSE
// create array of event (each per socket) and wait
}
void RecvSocketThread::WaitForPendingConnection_DoForAllSocketsInThePool()
{
// create array and set it for events associated with pending connect sockets
HANDLE* EventArray = NULL;
int counter = 0;
EventArray = new HANDLE[m_RecvSocketInfoPool.size()];
// add those event whose associated socket is still not connected
// and wait for FD_WRITE and FD_CLOSE. At the end of this function
// don't forget to switch them to FD_READ and FD_CLOSE
while (it != m_RecvSocketInfoPool.end())
{
RecvSocketInfo* recvSocketInfo = it->second;
if (!IsEventSet(recvSocketInfo->m_Connected, &retVal2))
{
::WSAEventSelect(recvSocketInfo->m_WorkerSocket, recvSocketInfo->m_Event, FD_WRITE | FD_CLOSE);
EventArray[counter++] = recvSocketInfo->m_Event;
}
++it;
}
if (counter)
{
DWORD indexSignaled = WaitForMultipleObjects(counter, EventArray, WaitAtLeastOneEvent, INFINITE);
// no matter what is further Wait doen't return for failed to connect socket
if (WAIT_OBJECT_0 <= indexSignaled &&
indexSignaled < (WAIT_OBJECT_0 + counter))
{
it = m_RecvSocketInfoPool.begin();
while (it != m_RecvSocketInfoPool.end())
{
RecvSocketInfo* recvSocketInfo = it->second;
if (IsEventSet(recvSocketInfo->m_Event, NULL))
{
rc = WSAEnumNetworkEvents(recvSocketInfo->m_WorkerSocket,
recvSocketInfo->m_Event, &networkEvents);
// Check recvSocketInfo->m_Event using WSAEnumnetworkevents
// for FD_CLOSE using FD_CLOSE_BIT
if ((networkEvents.lNetworkEvents & FD_CLOSE))
{
recvSocketInfo->m_FD_CLOSE_Recieved = 1;
*retVal = networkEvents.iErrorCode[FD_CLOSE_BIT];
}
if ((networkEvents.lNetworkEvents & FD_WRITE))
{
WSASetEvent(recvSocketInfo->m_Connected);
*retVal = networkEvents.iErrorCode[FD_WRITE_BIT];
}
}
++it;
}
}
// if error - DoClean, if FD_WRITE (socket is writable) check if m_Connected
// before do any sending
}
}
You will not receive an FD_CLOSE notification if connect() fails. You must subscribe to FD_CONNECT to detect that. This is clearly stated in the connect() documentation:
With a nonblocking socket, the connection attempt cannot be completed
immediately. In this case, connect will return SOCKET_ERROR, and
WSAGetLastError will return WSAEWOULDBLOCK. In this case, there are
three possible scenarios:
•Use the select function to determine the completion of the
connection request by checking to see if the socket is writeable.
•If the application is using WSAAsyncSelect to indicate interest in
connection events, then the application will receive an FD_CONNECT
notification indicating that the connect operation is complete
(successfully or not).
•If the application is using WSAEventSelect to indicate interest in
connection events, then the associated event object will be signaled
indicating that the connect operation is complete (successfully or
not).
The result code of connect() will be in the event's HIWORD(lParam) value when LOWORD(lParam) is FD_CONNECT. If the result code is 0, connect() was successful, otherwise it will be a WinSock error code.
If you call connect() and get a blocking notification you have to write more code to monitor for connect() completion (success or failure) via one of three methods as described here.
With a nonblocking socket, the connection attempt cannot be completed
immediately. In this case, connect will return SOCKET_ERROR, and
WSAGetLastError will return WSAEWOULDBLOCK. In this case, there are
three possible scenarios:
•Use the select function to determine the completion of the connection
request by checking to see if the socket is writeable.
•If the
application is using WSAAsyncSelect to indicate interest in connection
events, then the application will receive an FD_CONNECT notification
indicating that the connect operation is complete (successfully or
not).
•If the application is using WSAEventSelect to indicate interest
in connection events, then the associated event object will be
signaled indicating that the connect operation is complete
(successfully or not).
I think I need to start Receving thread once socket handle is created, but before connect is called. It is too late to create it after connect was called on asynchronous socket.
For synchronous socket those two calls createsocket() and connect() was just two consequitive lines. Does not work for non-blocking.
In this case at the beginning of receiving thread I need to check for FD_CONNECT and/or FD_WRITE in order be informed of connect attempt status.
I am building a Visual C++ WinSock TCP server using BindIoCompletionCallback, it works fine receiving and sending data, but I can't find a good way to detect timeout: SetSockOpt/SO_RCVTIMEO/SO_SNDTIMEO has no effect on nonblocking sockets, if the peer is not sending any data, the CompletionRoutine is not called at all.
I am thinking about using RegisterWaitForSingleObject with the hEvent field of OVERLAPPED, that might work but then CompletionRoutine is not needed at all, am I still using IOCP ? is there a performance concern if I use only RegisterWaitForSingleObject and not using BindIoCompletionCallback ?
Update: Code Sample:
My first try:
bool CServer::Startup() {
SOCKET ServerSocket = WSASocket(AF_INET, SOCK_STREAM, 0, NULL, 0, WSA_FLAG_OVERLAPPED);
WSAEVENT ServerEvent = WSACreateEvent();
WSAEventSelect(ServerSocket, ServerEvent, FD_ACCEPT);
......
bind(ServerSocket......);
listen(ServerSocket......);
_beginthread(ListeningThread, 128 * 1024, (void*) this);
......
......
}
void __cdecl CServer::ListeningThread( void* param ) // static
{
CServer* server = (CServer*) param;
while (true) {
if (WSAWaitForMultipleEvents(1, &server->ServerEvent, FALSE, 100, FALSE) == WSA_WAIT_EVENT_0) {
WSANETWORKEVENTS events = {};
if (WSAEnumNetworkEvents(server->ServerSocket, server->ServerEvent, &events) != SOCKET_ERROR) {
if ((events.lNetworkEvents & FD_ACCEPT) && (events.iErrorCode[FD_ACCEPT_BIT] == 0)) {
SOCKET socket = accept(server->ServerSocket, NULL, NULL);
if (socket != SOCKET_ERROR) {
BindIoCompletionCallback((HANDLE) socket, CompletionRoutine, 0);
......
}
}
}
}
}
}
VOID CALLBACK CServer::CompletionRoutine( __in DWORD dwErrorCode, __in DWORD dwNumberOfBytesTransfered, __in LPOVERLAPPED lpOverlapped ) // static
{
......
BOOL res = GetOverlappedResult(......, TRUE);
......
}
class CIoOperation {
public:
OVERLAPPED Overlapped;
......
......
};
bool CServer::Receive(SOCKET socket, PBYTE buffer, DWORD length, void* context)
{
if (connection != NULL) {
CIoOperation* io = new CIoOperation();
WSABUF buf = {length, (PCHAR) buffer};
DWORD flags = 0;
if ((WSARecv(Socket, &buf, 1, NULL, &flags, &io->Overlapped, NULL) != 0) && (GetLastError() != WSA_IO_PENDING)) {
delete io;
return false;
} else return true;
}
return false;
}
As I said, it works fine if the client is actually sending data to me, 'Receive' is not blocking, CompletionRoutine got called, data received, but here is one gotcha, if the client is not sending any data to me, how can I give up after a timeout ?
Since SetSockOpt/SO_RCVTIMEO/SO_SNDTIMEO wont help here, I think I should use the hEvent field in the OVERLAPPED stucture which will be signaled when the IO completes, but a WaitForSingleObject / WSAWaitForMultipleEvents on that will block the Receive call, and I want the Receive to always return immediately, so I used RegisterWaitForSingleObject and WAITORTIMERCALLBACK. it worked, the callback got called after the timeout, or, the IO completes, but now I have two callbacks for any single IO operation, the CompletionRoutine, and the WaitOrTimerCallback:
if the IO completed, they will be called simutaneously, if the IO is not completed, WaitOrTimerCallback will be called, then I call CancelIoEx, this caused the CompletionRoutine to be called with some ABORTED error, but here is a race condition, maybe the IO will be completed right before I cancel it, then ... blahblah, all in all its quite complicated.
Then I realized I dont actually need BindIoCompletionCallback and CompletionRoutine at all, and do everything from the WaitOrTimerCallback, it may work, but here is the interesting question, I wanted to build an IOCP-based Winsock server in the first place, and thought BindIoCompletionCallback is the easiest way to do that, using the threadpool provied by Windows itself, now I endup with a server without IOCP code at all ? is it still IOCP ? or should I forget BindIoCompletionCallback and build my own IOCP threadpool implementation ? why ?
What I did was to force the timeout/completion notifications to enter a critical section in the socket object. Once in, the winner can set a socket state variable and perform its action, whatever that might be. If the I/O completion gets in first, the I/O buffer array is processed in the normal way and any timeout is directed to restart by the state-machine. Similarly if the timeout gets in first, the I/O gets CancelIOEx'd and any later queued completion notification is discarded by the state-engine. Because of these possible 'late' notifications, I put released sockets onto a timeout queue and only recycle them onto the socket object pool after five minutes, in a similar way to how the TCP stack itself puts its sockets into 'TIME_WAIT'.
To do the timeouts, I have one thread that operates on FIFO delta-queues of timing-out objects, one queue for each timeout limit. The thread waits on an input queue for new objects with a timeout calculated from the smallest timeout-expiry-time of the objects at the head of the queues.
There were only a few timeouts used in the server, so I used queues fixed at compile-time. It would be fairly easy to add new queues or modify the timeout by sending appropriate 'command' messages to the thread input queue, mixed-in with the new sockets, but I didn't get that far.
Upon timeout, the thread called an event in the object which, in case of a socket, would enter the socket object CS-protected state-machine, (these was a TimeoutObject class which the socket descended from, amongst other things).
More:
I wait on the semaphore that controls the timeout thread input queue. If it's signaled, I get the new TimeoutObject from the input queue and add it to the end of whatever timeout queue it asks for. If the semaphore wait times out, I check the items at the heads of the timeout FIFO queues and recalculate their remaining interval by sutracting the current time from their timeout time. If the interval is 0 or negative, the timeout event gets called. While iterating the queues and their heads, I keep in a local the minimum remaining interval before the next timeout. Hwn all the head items in all the queues have non-zero remaining interval, I go back to waiting on the queue semaphore using the minimum remaining interval I have accumulated.
The event call returns an enumeration. This enumeration instructs the timeout thread how to handle an object whose event it's just fired. One option is to restart the timeout by recalcuating the timeout-time and pushing the object back onto its timeout queue at the end.
I did not use RegisterWaitForSingleObject() because it needed .NET and my Delphi server was all unmanaged, (I wrote my server a long time ago!).
That, and because, IIRC, it has a limit of 64 handles, like WaitForMultipleObjects(). My server had upwards of 23000 clients timing out. I found the single timeout thread and multiple FIFO queues to be more flexible - any old object could be timed out on it as long as it was descended from TimeoutObject - no extra OS calls/handles needed.
The basic idea is that, since you're using asynchronous I/O with the system thread pool, you shouldn't need to check for timeouts via events because you're not blocking any threads.
The recommended way to check for stale connections is to call getsockopt with the SO_CONNECT_TIME option. This returns the number of seconds that the socket has been connected. I know that's a poll operation, but if you're smart about how and when you query this value, it's actually a pretty good mechanism for managing connections. I explain below how this is done.
Typically I'll call getsockopt in two places: one is during my completion callback (so that I have a timestamp for the last time that an I/O completion occurred on that socket), and one is in my accept thread.
The accept thread monitors my socket backlog via WSAEventSelect and the FD_ACCEPT parameter. This means that the accept thread only executes when Windows determines that there are incoming connections that require accepting. At this time I enumerate my accepted sockets and query SO_CONNECT_TIME again for each socket. I subtract the timestamp of the connection's last I/O completion from this value, and if the difference is above a specified threshold my code deems the connection as having timed out.
I have a C++ pipe server app and a C# pipe client app communicating via Windows named pipe (duplex, message mode, wait/blocking in separate read thread).
It all works fine (both sending and receiving data via the pipe) until I try and write to the pipe from the client in response to a forms 'textchanged' event. When I do this, the client hangs on the pipe write call (or flush call if autoflush is off). Breaking into the server app reveals it's also waiting on the pipe ReadFile call and not returning.
I tried running the client write on another thread -- same result.
Suspect some sort of deadlock or race condition but can't see where... don't think I'm writing to the pipe simultaneously.
Update1: tried pipes in byte mode instead of message mode - same lockup.
Update2: Strangely, if (and only if) I pump lots of data from the server to the client, it cures the lockup!?
Server code:
DWORD ReadMsg(char* aBuff, int aBuffLen, int& aBytesRead)
{
DWORD byteCount;
if (ReadFile(mPipe, aBuff, aBuffLen, &byteCount, NULL))
{
aBytesRead = (int)byteCount;
aBuff[byteCount] = 0;
return ERROR_SUCCESS;
}
return GetLastError();
}
DWORD SendMsg(const char* aBuff, unsigned int aBuffLen)
{
DWORD byteCount;
if (WriteFile(mPipe, aBuff, aBuffLen, &byteCount, NULL))
{
return ERROR_SUCCESS;
}
mClientConnected = false;
return GetLastError();
}
DWORD CommsThread()
{
while (1)
{
std::string fullPipeName = std::string("\\\\.\\pipe\\") + mPipeName;
mPipe = CreateNamedPipeA(fullPipeName.c_str(),
PIPE_ACCESS_DUPLEX,
PIPE_TYPE_MESSAGE | PIPE_READMODE_MESSAGE | PIPE_WAIT,
PIPE_UNLIMITED_INSTANCES,
KTxBuffSize, // output buffer size
KRxBuffSize, // input buffer size
5000, // client time-out ms
NULL); // no security attribute
if (mPipe == INVALID_HANDLE_VALUE)
return 1;
mClientConnected = ConnectNamedPipe(mPipe, NULL) ? TRUE : (GetLastError() == ERROR_PIPE_CONNECTED);
if (!mClientConnected)
return 1;
char rxBuff[KRxBuffSize+1];
DWORD error=0;
while (mClientConnected)
{
Sleep(1);
int bytesRead = 0;
error = ReadMsg(rxBuff, KRxBuffSize, bytesRead);
if (error == ERROR_SUCCESS)
{
rxBuff[bytesRead] = 0; // terminate string.
if (mMsgCallback && bytesRead>0)
mMsgCallback(rxBuff, bytesRead, mCallbackContext);
}
else
{
mClientConnected = false;
}
}
Close();
Sleep(1000);
}
return 0;
}
client code:
public void Start(string aPipeName)
{
mPipeName = aPipeName;
mPipeStream = new NamedPipeClientStream(".", mPipeName, PipeDirection.InOut, PipeOptions.None);
Console.Write("Attempting to connect to pipe...");
mPipeStream.Connect();
Console.WriteLine("Connected to pipe '{0}' ({1} server instances open)", mPipeName, mPipeStream.NumberOfServerInstances);
mPipeStream.ReadMode = PipeTransmissionMode.Message;
mPipeWriter = new StreamWriter(mPipeStream);
mPipeWriter.AutoFlush = true;
mReadThread = new Thread(new ThreadStart(ReadThread));
mReadThread.IsBackground = true;
mReadThread.Start();
if (mConnectionEventCallback != null)
{
mConnectionEventCallback(true);
}
}
private void ReadThread()
{
byte[] buffer = new byte[1024 * 400];
while (true)
{
int len = 0;
do
{
len += mPipeStream.Read(buffer, len, buffer.Length);
} while (len>0 && !mPipeStream.IsMessageComplete);
if (len==0)
{
OnPipeBroken();
return;
}
if (mMessageCallback != null)
{
mMessageCallback(buffer, len);
}
Thread.Sleep(1);
}
}
public void Write(string aMsg)
{
try
{
mPipeWriter.Write(aMsg);
mPipeWriter.Flush();
}
catch (Exception)
{
OnPipeBroken();
}
}
If you are using separate threads you will be unable to read from the pipe at the same time you write to it. For example, if you are doing a blocking read from the pipe then a subsequent blocking write (from a different thread) then the write call will wait/block until the read call has completed and in many cases if this is unexpected behavior your program will become deadlocked.
I have not tested overlapped I/O, but it MAY be able to resolve this issue. However, if you are determined to use synchronous calls then the following models below may help you to solve the problem.
Master/Slave
You could implement a master/slave model in which the client or the server is the master and the other end only responds which is generally what you will find the MSDN examples to be.
In some cases you may find this problematic in the event the slave periodically needs to send data to the master. You must either use an external signaling mechanism (outside of the pipe) or have the master periodically query/poll the slave or you can swap the roles where the client is the master and the server is the slave.
Writer/Reader
You could use a writer/reader model where you use two different pipes. However, you must associate those two pipes somehow if you have multiple clients since each pipe will have a different handle. You could do this by having the client send a unique identifier value on connection to each pipe which would then let the server associate the two pipes. This number could be the current system time or even a unique identifier that is global or local.
Threads
If you are determined to use the synchronous API you can use threads with the master/slave model if you do not want to be blocked while waiting for a message on the slave side. You will however want to lock the reader after it reads a message (or encounters the end of a series of message) then write the response (as the slave should) and finally unlock the reader. You can lock and unlock the reader using locking mechanisms that put the thread to sleep as these would be most efficient.
Security Problem With TCP
The loss going with TCP instead of named pipes is also the biggest possible problem. A TCP stream does not contain any security natively. So if security is a concern you will have to implement that and you have the possibility of creating a security hole since you would have to handle authentication yourself. The named pipe can provide security if you properly set the parameters. Also, to note again more clearly: security is no simple matter and generally you will want to use existing facilities that have been designed to provide it.
I think you may be running into problems with named pipes message mode. In this mode, each write to the kernel pipe handle constitutes a message. This doesn't necessarily correspond with what your application regards a Message to be, and a message may be bigger than your read buffer.
This means that your pipe reading code needs two loops, the inner reading until the current [named pipe] message has been completely received, and the outer looping until your [application level] message has been received.
Your C# client code does have a correct inner loop, reading again if IsMessageComplete is false:
do
{
len += mPipeStream.Read(buffer, len, buffer.Length);
} while (len>0 && !mPipeStream.IsMessageComplete);
Your C++ server code doesn't have such a loop - the equivalent at the Win32 API level is testing for the return code ERROR_MORE_DATA.
My guess is that somehow this is leading to the client waiting for the server to read on one pipe instance, whilst the server is waiting for the client to write on another pipe instance.
It seems to me that what you are trying to do will rather not work as expected.
Some time ago I was trying to do something that looked like your code and got similar results, the pipe just hanged
and it was difficult to establish what had gone wrong.
I would rather suggest to use client in very simple way:
CreateFile
Write request
Read answer
Close pipe.
If you want to have two way communication with clients which are also able to receive unrequested data from server you should
rather implement two servers. This was the workaround I used: here you can find sources.