I wrote a very simple code to set up a "server" that creates a named pipe and waits for a client to connect. As soon as the client opens the pipe, the server sends its data (a block of about 10mb) and the client is supposed to read it and close the connection.
The real catch now is: When the pipe is working with local names (\.\pipe\xxx) it does send all the data without any problem but if i change the path to a network name (\computer\pipe\xxx) it changes behavior and client can only read about 65000~ bytes, but it does not complete read operation even when i loop it (i suppose it breaks in 65k blocks to send over network since i'm using a network name, it happens even locally). ReadFile reads the 65k block and returns TRUE, if i try to force ReadFile again in the pipe it reads 0 bytes.
The flags i'm using to create the pipe are PIPE_ACCESS_DUPLEX, FILE_FLAG_WRITE_THROUGH, PIPE_TYPE_BYTE, PIPE_READMODE_BYTE, PIPE_WAIT, PIPE_ACCEPT_REMOTE_CLIENTS
Here is a piece of what the code should look like (the code is somewhere else and i can't access it right now but as i said before, it is as simple as it gets)
lPipe := CreateFileA('\\.\pipe\test', GENERIC_READ or GENERIC_WRITE, FILE_SHARE_READ or FILE_SHARE_WRITE, nil, OPEN_EXISTING, 0, 0);
ReadFile(lPipe, lMemoryStream.Memory^, 1024*1024*15, lBytesRead, nil);
CloseHandle(lPipe);
From the MSDN documentation for WriteFileEx:
Pipe write operations across a network are limited to 65,535 bytes per write. For more information regarding pipes, see the Remarks section.
To get past this, you'll have to set up the server to send the data in chunks.
Related
I think I need some clarification on how to read from a named pipe and have it return immediately, data or not. What I am seeing is ReadFile fails, as expected, but GetLastError returns either ERROR_IO_PENDING or ERROR_PIPE_NOT_CONNECTED, and it does this until my surrounding code times out. I get these errors EVEN THOUGH THE DATA HAS IN FACT ARRIVED. I know this by checking my read buffer and seeing what I expect. And the pipe keeps working. I suspect I am not using the overlapped structure correctly, I'm just setting all fields to zero. My code looks like this:
gPipe = CreateFile(gPipename, GENERIC_READ | GENERIC_WRITE, 0, NULL, OPEN_EXISTING, FILE_FLAG_OVERLAPPED, NULL);
pMode = PIPE_READMODE_MESSAGE;
bret = SetNamedPipeHandleState(gPipe, &pMode, NULL, NULL);
OVERLAPPED ol;
memset(&ol, 0, sizeof(OVERLAPPED));
// the following inside a loop that times out after a period
bret = ReadFile(gPipe, &tmostat, sizeof(TMO64STAT), NULL, &ol);
if (bret) break;
err = GetLastError();
// seeing err == ERROR_IO_PENDING or ERROR_PIPE_NOT_CONNECTED
So I can do what I want by ignoring the errors and checking for arrived data, but it troubles me. Any idea why I am getting this behavior?
Windows OVERLAPPED I/O doesn't work like the non-blocking flag on other OSes (For example on Linux, the closest equivalent is aio_*() API, not FIONBIO)
With OVERLAPPED I/O, the operation hasn't failed, it proceeds in the background. But you are never checking on it... you just try again. There's a queue of pending operations, you're always starting new ones, never checking on the old ones.
Fill in the hEvent field in the OVERLAPPED structure, and use it to detect when the operation completes. Then call GetOverlappedResult() to get the number of bytes actually transferred.
Another important note -- the OS owns the OVERLAPPED structure and the buffer until the operation completes, you must take care to make sure these stay valid and not deallocate them or use them for any other operation until you confirm that the first operation completed.
Note that there is an actual non-blocking mode for Win32 pipes, but Microsoft strongly recommends against using it:
The nonblocking-wait mode is supported for compatibility with Microsoft LAN Manager version 2.0. This mode should not be used to achieve overlapped input and output (I/O) with named pipes. Overlapped I/O should be used instead, because it enables time-consuming operations to run in the background after the function returns.
Named Pipe Type, Read, and Wait Modes
I use Lazarus 1.2.2 and Freepascal 2.6.4.
I have a program called QuickHash that hashes files and, when I run it on Linux, it can be used to hash physcial disks too (/dev/sdXX). However, I'd like to add the ability to the Windows version.
I gather that to access physcial devices, like disks, one has to use CreateFile. Specifically, CreateFileW.
So, the user clicks a button which scans the computer for disks and lists them in a List Box. The one the user double clicks is then parsed (ListBox.GetSelectedText) for the string '\.\PhyscialDiskX' and that is assigned to a string variable,
strDiskID := getDiskID(Listbox.GetSelectedText);
That works fine.
I then try to create a handle to that disk :
hSelectedDisk := Windows.CreateFileW(PWideChar(strDiskID), GENERIC_READ, FILE_SHARE_READ, nil, OPEN_EXISTING, FILE_FLAG_RANDOM_ACCESS, 0);
Based on this article, specifcally "You must use both the CreateFile() FILE_SHARE_READ and FILE_SHARE_WRITE flags to gain access to the drive" I have also tried the other two combinations below :
hSelectedDisk := CreateFileW(PWideChar(strDiskID), GENERIC_READ, FILE_SHARE_READ AND FILE_SHARE_WRITE, nil, OPEN_EXISTING, FILE_FLAG_RANDOM_ACCESS, 0);
hSelectedDisk := CreateFileW(PWideChar(strDiskID), GENERIC_READ, FILE_SHARE_READ OR FILE_SHARE_WRITE, nil, OPEN_EXISTING, FILE_FLAG_RANDOM_ACCESS, 0);
All three successfully assign a handle. But, the top syntax and the bottom syntax eventually generate an error (explained below). The middle option immediately returns the default initialisation hash for a zero byte file, i.e. DA39... for SHA1.
My problem is I am unable to pass that handle (which is an integer) to the SHA1File and MD5FILE functions of the Freepascal md5 and SHA1 units. They expect a filename, which has to be a string.
So, if I pass it the strDiskID ('\.\PhyscialDiskX') (which defeats the objects of assigning a handle at all) I do get disk activity and the program appears to be working.
strDiskHashValue := SHA1Print(SHA1File(strDiskID));
ShowMessage(strDiskHashValue);
CloseHandle(hSelectedDisk);
But even when run on really small disks like a 500Mb USB drive, it takes many minutes and eventually returns "Run Error 1117" which according to this means
"ERROR_IO_DEVICE
1117 (0x45D)
The request could not be performed because of an I/O device error."
However, I have tried it on several working disks and the error continues.
So, my question, ultimately, is how on earth do I pass that successfully assigned THandle to the hashing functions? (UI have also asked the question at the Lazarus forums but sometimes I get answers here from members who don't see the threads there)
You aren't going to be able to pass volume handles to functions that aren't expecting volume handles. These are very special handles with quite stringent requirements on their use. Not the least of which is that you must read blocks that are sector aligned, and whose sizes are multiples of the sector size.
So the solution is for you to take charge of reading the data. Read it into a buffer, and then pass that buffer to the hashing library. This means that you'll need a hashing library that can be called repeatedly to process new data. All comprehensive hashing libraries will offer such functionality.
The share mode flags are combined with bitwise or:
FILE_SHARE_READ or FILE_SHARE_WRITE
I would create the handle like this:
hSelectedDisk := CreateFileW(PWideChar(strDiskID), FILE_READ_DATA,
FILE_SHARE_READ or FILE_SHARE_WRITE, nil, OPEN_EXISTING, 0, 0);
First of all I would concentrate on reading the content of the volume. Once you can do that, hashing will be routine.
It seems from the comments that you are having some trouble writing the hashing code. First of all you need to allocate a buffer that is a multiple of the sector size:
var
Buffer: Pointer;
....
GetMem(Buffer, BufferSize);
Use IOCTL_DISK_GET_DRIVE_GEOMETRY to find out the sector size. And take note of this text from the documentation:
To read or write to the last few sectors of the volume, you must call DeviceIoControl and specify FSCTL_ALLOW_EXTENDED_DASD_IO. This signals the file system driver not to perform any I/O boundary checks on partition read or write calls. Instead, boundary checks are performed by the device driver.
Now that you have a buffer, you can read and hash the content.
var
ctx: TSHA1Context;
Digest: TSHA1Digest;
BytesRead: DWORD;
....
SHA1Init(ctx);
repeat
if not ReadFile(hSelectedDisk, Buffer^, BufferSize, BytesRead, nil) then
// handle error, raise exception
SHA1Update(ctx, Buffer^, BytesRead);
until BytesRead < BufferSize;
SHA1Final(ctx, Digest);
I've not attempted to compile or test this code. It's not meant to be complete or comprehensive. It is just intended to show you how to tackle the problem.
Starting with the ndisprot sample from Microsoft I try to write a NDIS protocol driver. From User space I try to read and write to the device simultaneous (out of two threads). Since I don't receive any packets, the ReadFile system call blocks. I'm not able to complete a WriteFile system call in this state.
CHAR NdisProtDevice[] = "\\\\.\\\\NDISprot";
CHAR * pNdisProtDevice = &NdisProtDevice[0];
this.iHandle = CreateFile(pNdisProtDevice,
GENERIC_WRITE | GENERIC_READ, 0, 0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);
// Blocks, because no frames arrive
bSuccess = (BOOLEAN)ReadFile(Handle,
(LPVOID)pReadBuf,
PacketLength,
&BytesRead,
NULL);
...
// Called some seconds later from another thread, while ReadFile still blocking...
bSuccess = (BOOLEAN)WriteFile(Handle,
pWriteBuf,
PacketLength,
&BytesWritten,
NULL);
I added some debug messages and discovered that the driver function associated with IRP_MJ_WRITE (NdisprotWrite) gets not even called! Something between the user space application and the driver blocks concurrent access to the device \Device\NDISprot.
How can I concurrent Read and Write to the file?
By default, you can only have one outstanding I/O request per usermode handle. Either open multiple handles, or open your one handle with FILE_FLAG_OVERLAPPED. (Once you use FILE_FLAG_OVERLAPPED, you also generally need to use OVERLAPPED structures - make sure you've got the gist of it by skimming this and this.)
in one process, i called createfile with GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, OPEN_ALWAYS, and FILE_ATTRIBUTE_NORMAL as the params.
then i called LockFileEx on the whole file. i acquired an exclusive lock, and locked from range 0 to UINT_MAX.
after that, in another process, i called ::CreateFileW(path.c_str(), perms, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
and it erased the contents of the file. shouldn't it not be able to do that while my other prcoess has the file locked (exclusively)?
i want other processes to be able to get handles to the file (which is why i used the file_share flags), but i assumed they couldnt change the file at all while another process locked it, and so i was doing something like
createfile
lockfileex
and having it block on lockfileex until the other file releases the lock. is it just me or is msft's behavior here wrong?
It is debatable whether Windows should honor file locks when you attempt to open a file with CREATE_ALWAYS, but it appears from your notes that it does not. The best way to prevent other processes from writing to a file that you have open is not to specify FILE_SHARE_WRITE in your exclusive process.
However, this won't give you the waiting behavior that you would get from the file lock implementation without polling. There is no way to open a file with exclusive access and have any other files that are attempting to open it wait until the process with exclusive access relinquishes that access.
If you have access to the source of all of the processes involved, then you could just have the processes that are attempting to truncate the file first call LockFileEx on a region, and then call SetFileSize after acquiring the lock.
As a side note, only locking the region from 0 bytes to UINT_MAX won't provide mutual exclusion if, say, a process opens the file and attempts to write to the location at UINT_MAX + 1 (just past the 4GB point).
A Win32 application (the "server") is sending a continuous stream of data over a named pipe. GetNamedPipeInfo() tells me that input and output buffer sizes are automatically allocated as needed. The pipe is operating in byte mode (although it is sending data units that are bigger than 1 byte (doubles, to be precise)).
Now, my question is this: Can I somehow verify that my application (the "client") is not missing any data when reading from the pipe? I know that those read/write operations are buffered, but I suppose the buffers will not grow indefinitely if the client doesn't fetch the data quickly enough. How do I know if I missed something? Does the server (or the pipe?) silently discard data that is not read in time by the client?
BTW, can I rely on proper alignment of the data the client reads using ReadFile()? As far as I understood, ReadFile() may return with less bytes read than specified, i.e. NumberOfBytesRead <= NumberOfBytesToRead. Do I have to check every time that NumberOfBytesRead is a multiple of sizeof(double)?
The write operation will block if there is no more room in the pipe's buffers. This is from my (old) copy of the SDK manual:
When an application uses the WriteFile
function to write to a pipe, the write
operation may not finish if the pipe
buffer is full. The write operation is
completed when a read operation (using
the ReadFile function) makes more
buffer space available.
Sorry, didn't find out how to comment on your post, Neil.
The write operation will block if there is no more room in the pipe's buffers.
I just discovered that Sysinternals' FileMon can also monitor pipe operations. For testing purposes I connected the client to the named pipe and did no read operations, just waiting. The server writes a few hundred kB to the pipe every 4--5 seconds, even though nobody is fetching the data from the pipe on the client side. No blocking write operation ... And so far no limits in buffer-size seem to have been reached.
This is either a very big buffer ... or the server does some magic additional to just using WriteFile() and waiting for the client to read.