I think I need some clarification on how to read from a named pipe and have it return immediately, data or not. What I am seeing is ReadFile fails, as expected, but GetLastError returns either ERROR_IO_PENDING or ERROR_PIPE_NOT_CONNECTED, and it does this until my surrounding code times out. I get these errors EVEN THOUGH THE DATA HAS IN FACT ARRIVED. I know this by checking my read buffer and seeing what I expect. And the pipe keeps working. I suspect I am not using the overlapped structure correctly, I'm just setting all fields to zero. My code looks like this:
gPipe = CreateFile(gPipename, GENERIC_READ | GENERIC_WRITE, 0, NULL, OPEN_EXISTING, FILE_FLAG_OVERLAPPED, NULL);
pMode = PIPE_READMODE_MESSAGE;
bret = SetNamedPipeHandleState(gPipe, &pMode, NULL, NULL);
OVERLAPPED ol;
memset(&ol, 0, sizeof(OVERLAPPED));
// the following inside a loop that times out after a period
bret = ReadFile(gPipe, &tmostat, sizeof(TMO64STAT), NULL, &ol);
if (bret) break;
err = GetLastError();
// seeing err == ERROR_IO_PENDING or ERROR_PIPE_NOT_CONNECTED
So I can do what I want by ignoring the errors and checking for arrived data, but it troubles me. Any idea why I am getting this behavior?
Windows OVERLAPPED I/O doesn't work like the non-blocking flag on other OSes (For example on Linux, the closest equivalent is aio_*() API, not FIONBIO)
With OVERLAPPED I/O, the operation hasn't failed, it proceeds in the background. But you are never checking on it... you just try again. There's a queue of pending operations, you're always starting new ones, never checking on the old ones.
Fill in the hEvent field in the OVERLAPPED structure, and use it to detect when the operation completes. Then call GetOverlappedResult() to get the number of bytes actually transferred.
Another important note -- the OS owns the OVERLAPPED structure and the buffer until the operation completes, you must take care to make sure these stay valid and not deallocate them or use them for any other operation until you confirm that the first operation completed.
Note that there is an actual non-blocking mode for Win32 pipes, but Microsoft strongly recommends against using it:
The nonblocking-wait mode is supported for compatibility with Microsoft LAN Manager version 2.0. This mode should not be used to achieve overlapped input and output (I/O) with named pipes. Overlapped I/O should be used instead, because it enables time-consuming operations to run in the background after the function returns.
Named Pipe Type, Read, and Wait Modes
Related
What would cause WriteFile to return error 38 (ERROR_HANDLE_EOF, Reached the end of the file)? The "file" in this case is a mailslot. The way my program works is I have a process (running as a Windows service) that creates multiple child processes. Each child opens a mailslot of the same name in order to send status information back to its parent. In my small scale testing this works fine, but I am seeing cases where when I have several processes
running (like 16) I am getting this error. The code below shows how I am opening and writing to the mailslot in the child process.
Is it perhaps because the parent is not reading the mailslot fast enough? Is there a way to increase capacity of a mailslot so that end of file never gets reached? I really don't see how a mailslot can get full anyway, as long
as there is disk space (which there is plenty of).
char gLocalSlotName[256]="\\\\.\\mailslot\\TMAgentSlot-ComputerName";
gAgentSlot = CreateFile(gLocalSlotName, GENERIC_WRITE, FILE_SHARE_READ, (LPSECURITY_ATTRIBUTES) NULL,
OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, (HANDLE) NULL);
fResult = WriteFile(gAgentSlot, (char *)&ProcStat, sizeof(PROCSTAT), &cbWritten, (LPOVERLAPPED) NULL);
if (!fResult) {
derr = GetLastError();
printf("WriteFile error=%d", derr);
}
WriteFile is thin shell over NtWriteFile. if NtWriteFile return error NTSTATUS - it will be converted to its equivalent win32 error code (via RtlNtStatusToDosError) and WriteFile return false. win32 error code you can got via GetLastError(). however original NTSTATUS you can got via RtlGetLastNtStatus() exported by ntdll.dll api. problem with win32 errors codes - some time several different NTSTATUS values converted to the same win32 error.
in case ERROR_HANDLE_EOF - 2 different NTSTATUS converted to it:
STATUS_END_OF_FILE and STATUS_FILE_FORCED_CLOSED. the STATUS_END_OF_FILE never (look like) returned by msfs.sys (driver which handle mailslots). from another side - STATUS_FILE_FORCED_CLOSED (The specified file has been closed by another process.) can be returned when you write data to mailslot (by msfs.MsCommonWrite) if server end of the mailslot (end which you create via CreateMailslot call) already closed.
formally when last server handle was closed - all connecting clients marked as in closing state (inside MsFsdCleanup) and then if you call WriteFile for such client - the STATUS_FILE_FORCED_CLOSED is returned.
so -
What causes WriteFile to return error 38 (ERROR_HANDLE_EOF)?
the server process by some reason close self mailslot handle. you need search in this direction - when and why you close mailsot handle in parent process
I wrote a very simple code to set up a "server" that creates a named pipe and waits for a client to connect. As soon as the client opens the pipe, the server sends its data (a block of about 10mb) and the client is supposed to read it and close the connection.
The real catch now is: When the pipe is working with local names (\.\pipe\xxx) it does send all the data without any problem but if i change the path to a network name (\computer\pipe\xxx) it changes behavior and client can only read about 65000~ bytes, but it does not complete read operation even when i loop it (i suppose it breaks in 65k blocks to send over network since i'm using a network name, it happens even locally). ReadFile reads the 65k block and returns TRUE, if i try to force ReadFile again in the pipe it reads 0 bytes.
The flags i'm using to create the pipe are PIPE_ACCESS_DUPLEX, FILE_FLAG_WRITE_THROUGH, PIPE_TYPE_BYTE, PIPE_READMODE_BYTE, PIPE_WAIT, PIPE_ACCEPT_REMOTE_CLIENTS
Here is a piece of what the code should look like (the code is somewhere else and i can't access it right now but as i said before, it is as simple as it gets)
lPipe := CreateFileA('\\.\pipe\test', GENERIC_READ or GENERIC_WRITE, FILE_SHARE_READ or FILE_SHARE_WRITE, nil, OPEN_EXISTING, 0, 0);
ReadFile(lPipe, lMemoryStream.Memory^, 1024*1024*15, lBytesRead, nil);
CloseHandle(lPipe);
From the MSDN documentation for WriteFileEx:
Pipe write operations across a network are limited to 65,535 bytes per write. For more information regarding pipes, see the Remarks section.
To get past this, you'll have to set up the server to send the data in chunks.
Starting with the ndisprot sample from Microsoft I try to write a NDIS protocol driver. From User space I try to read and write to the device simultaneous (out of two threads). Since I don't receive any packets, the ReadFile system call blocks. I'm not able to complete a WriteFile system call in this state.
CHAR NdisProtDevice[] = "\\\\.\\\\NDISprot";
CHAR * pNdisProtDevice = &NdisProtDevice[0];
this.iHandle = CreateFile(pNdisProtDevice,
GENERIC_WRITE | GENERIC_READ, 0, 0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);
// Blocks, because no frames arrive
bSuccess = (BOOLEAN)ReadFile(Handle,
(LPVOID)pReadBuf,
PacketLength,
&BytesRead,
NULL);
...
// Called some seconds later from another thread, while ReadFile still blocking...
bSuccess = (BOOLEAN)WriteFile(Handle,
pWriteBuf,
PacketLength,
&BytesWritten,
NULL);
I added some debug messages and discovered that the driver function associated with IRP_MJ_WRITE (NdisprotWrite) gets not even called! Something between the user space application and the driver blocks concurrent access to the device \Device\NDISprot.
How can I concurrent Read and Write to the file?
By default, you can only have one outstanding I/O request per usermode handle. Either open multiple handles, or open your one handle with FILE_FLAG_OVERLAPPED. (Once you use FILE_FLAG_OVERLAPPED, you also generally need to use OVERLAPPED structures - make sure you've got the gist of it by skimming this and this.)
According to MSDN, ReadFile can read data 2 different ways: synchronously and asynchronously.
I need the second one. The folowing code demonstrates usage with OVERLAPPED struct:
#include <windows.h>
#include <stdio.h>
#include <time.h>
void Read()
{
HANDLE hFile = CreateFileA("c:\\1.avi", GENERIC_READ, 0, NULL, OPEN_EXISTING, FILE_FLAG_OVERLAPPED, NULL);
if ( hFile == INVALID_HANDLE_VALUE )
{
printf("Failed to open the file\n");
return;
}
int dataSize = 256 * 1024 * 1024;
char* data = (char*)malloc(dataSize);
memset(data, 0xFF, dataSize);
OVERLAPPED overlapped;
memset(&overlapped, 0, sizeof(overlapped));
printf("reading: %d\n", time(NULL));
BOOL result = ReadFile(hFile, data, dataSize, NULL, &overlapped);
printf("sent: %d\n", time(NULL));
DWORD bytesRead;
result = GetOverlappedResult(hFile, &overlapped, &bytesRead, TRUE); // wait until completion - returns immediately
printf("done: %d\n", time(NULL));
CloseHandle(hFile);
}
int main()
{
Read();
}
On Windows XP output is:
reading: 1296651896
sent: 1296651896
done: 1296651899
It means that ReadFile didn't block and returned imediatly at the same second, whereas reading process continued for 3 seconds. It is normal async reading.
But on windows 7 and windows 2008 I get following results:
reading: 1296661205
sent: 1296661209
done: 1296661209.
It is a behavior of sync reading.
MSDN says that async ReadFile sometimes can behave as sync (when the file is compressed or encrypted for example). But the return value in this situation should be TRUE and GetLastError() == NO_ERROR.
On Windows 7 I get FALSE and GetLastError() == ERROR_IO_PENDING. So WinApi tells me that it is an async call, but when I look at the test I see that it is not!
I'm not the only one who found this "bug": read the comment on ReadFile MSDN page.
So what's the solution? Does anybody know? It is been 14 months after Denis found this strange behavior.
I don't know the size of the "c:\1.avi" file but the size of the buffer you give to Windows (256M!) is probably big enough to hold the file. So windows decides to read the whole file and put it in the buffer the way it likes. You don't say to windows "I want async", you say "I know how to handle async".
Just change the buffer size say 1024, and your program will behave exactly the same, but read only 1024 bytes (and return ERROR_IO_PENDING as well).
In general, you do asynchronous because you want to do something else during the operation. Look at the sample here: Testing for the End of a File, as it demonstrate an async ReadFile. If you change the sample's buffer and set it to a big value, it should behave exactly like yours.
PS: I suggest you don't rely on time samples to check things, use return codes and events
According to this, I would suspect that it should return TRUE in your case. But it may also be that the completion modes default settings are different on Win7/Win2k8.
Try setting a different mode with SetFileCompletionNotificationModes().
Have you tried to use an event as #Simon Mourier suggested ?. I know that the documentation says that the event is not required, but if you see the example in links provided by #Simon Mourier, it is using an event for asynchronous read.
Windows7/Server2008 have different behavior to resolve a race condition that can occurn in GetOverlappedResultEx. When you compile for these OS's Windows detects this and uses different behavior. I find this wicked confusing.
Here is a link:
http://msdn.microsoft.com/en-us/library/dd371711(VS.85).aspx
I'm sure you've read this many times in the past, but some of the text has changed since Win7 - esp the hEvent field in the OVERLAPPED struct,
http://msdn.microsoft.com/en-us/library/ms684342(v=VS.85).aspx
Functions such as
GetOverlappedResult and the
synchronization wait functions reset
auto-reset events to the nonsignaled
state. Therefore, you should use a
manual reset event; if you use an
auto-reset event, your application can
stop responding if you wait for the
operation to complete and then call
GetOverlappedResult with the bWait
parameter set to TRUE.
could you do an experiment - please allocate a manual reset event in your OVERLAPPED struct instead of a auto reset event? (I dont see the allocation in your snippit - dont forget to create the event and to set 'hEvent' after zeroing the struct)
This probably has something to do with caching. Try to open the file non-cached (FILE_FLAG_NO_BUFFERING)
EDIT
This is actually documented in the MSDN documentation for ReadFile:
Note If a file or device is opened
for asynchronous I/O, subsequent calls
to functions such as ReadFile using
that handle generally return
immediately, but can also behave
synchronously with respect to blocked
execution. For more information see
http://support.microsoft.com/kb/156932.
I'm spawning a process from Win32 using CreateProcess, setting the hStdOutput and hStdError properties of STARTUPINFO to pipe handles created with CreatePipe. I've got two threads reading the pipes, waiting for data to become available (or the process to complete, at which point it checks that there is no data left before terminating the thread).
As data becomes available, I write the output out to effectively a big textbox.
What's happening is the output is being buffered, so a slow running process just gets chunks of data thrown at the text box, but not "as it happens".
I'm not sure if it's the pipe that's doing the buffering, or something to do with the redirection.
Is there any way to either set the pipe to be unbuffered, or start the process in such a way that the stdout is sent as soon as possible?
I'm testing with a test app that prints lines one second apart
Here is line one
(waits one second)
Here is line two
(waits one second)
... etc
The buffering is probably in the C runtime (printf etc) and there is not much you can do about it (IIRC it does a isatty() check to determine a buffering strategy)
In my case the buffering was in the output of the client (as #Anders wrote), which uses normal printf. Maybe this also depends on the implementation of the C runtime (Visual Studio 2019), maybe the runtime detects 'not a console' and enables buffering.
So I disabled the buffering with this call
setvbuf(stdout, (char*)NULL, _IONBF, 0); in my client, now I get the output immediately in the pipe in the server.
Just for completeness: Here's how I read the pipe in the server
HANDLE in;
CreatePipe(&in, &startup.hStdOutput, &sa, 0); // Pipe for stdout of the child
...
char buffer[16384];
DWORD read, total;
while (PeekNamedPipe(in, NULL, 0, &read, &total, NULL))
{
if (total == 0)
{
if (WaitForSingleObject(info.hProcess, 0) == WAIT_OBJECT_0)
{
if (PeekNamedPipe(in, NULL, 0, &read, &total, NULL) && total == 0)
break;
continue;
}
Sleep(10);
continue;
}
if (total > sizeof(buffer))
total = sizeof(buffer);
ReadFile(in, buffer, total, &read, NULL);
...
}
There's SetNamedPipeHandleState, but it only controls buffering for remote pipes, not when both ends are on the same computer.
It seems to me you can solve the problem if you set the hStdOutput and hStdError of STARTUPINFO not to pipe handles created with CreatePipe, but instead of that you create a named pipes (with CallNamedPipe function exactly like you used if before also using SECURITY_ATTRIBUTES with bInheritHandle = TRUE, see http://msdn.microsoft.com/en-us/library/aa365782.aspx) and then open there by name with respect of CreateFile using FILE_FLAG_WRITE_THROUGH flag. Like you can read on the MSDN (http://msdn.microsoft.com/en-us/library/aa365592.aspx):
The pipe client can use CreateFile to
enable overlapped mode by specifying
FILE_FLAG_OVERLAPPED or to enable
write-through mode by specifying
FILE_FLAG_WRITE_THROUGH.
So just reopen the pipe with respect of CreateFile using FILE_FLAG_WRITE_THROUGH flag and set the handle/handles to hStdOutput and hStdError of STARTUPINFO.