A Win32 application (the "server") is sending a continuous stream of data over a named pipe. GetNamedPipeInfo() tells me that input and output buffer sizes are automatically allocated as needed. The pipe is operating in byte mode (although it is sending data units that are bigger than 1 byte (doubles, to be precise)).
Now, my question is this: Can I somehow verify that my application (the "client") is not missing any data when reading from the pipe? I know that those read/write operations are buffered, but I suppose the buffers will not grow indefinitely if the client doesn't fetch the data quickly enough. How do I know if I missed something? Does the server (or the pipe?) silently discard data that is not read in time by the client?
BTW, can I rely on proper alignment of the data the client reads using ReadFile()? As far as I understood, ReadFile() may return with less bytes read than specified, i.e. NumberOfBytesRead <= NumberOfBytesToRead. Do I have to check every time that NumberOfBytesRead is a multiple of sizeof(double)?
The write operation will block if there is no more room in the pipe's buffers. This is from my (old) copy of the SDK manual:
When an application uses the WriteFile
function to write to a pipe, the write
operation may not finish if the pipe
buffer is full. The write operation is
completed when a read operation (using
the ReadFile function) makes more
buffer space available.
Sorry, didn't find out how to comment on your post, Neil.
The write operation will block if there is no more room in the pipe's buffers.
I just discovered that Sysinternals' FileMon can also monitor pipe operations. For testing purposes I connected the client to the named pipe and did no read operations, just waiting. The server writes a few hundred kB to the pipe every 4--5 seconds, even though nobody is fetching the data from the pipe on the client side. No blocking write operation ... And so far no limits in buffer-size seem to have been reached.
This is either a very big buffer ... or the server does some magic additional to just using WriteFile() and waiting for the client to read.
Related
Wonder if anyone knows the internal design of WriteFile() (Storage Team Here?) with overlapped IO for file on on a disk drive/file system. Clearly when using the system buffer and standard synchronous WriteFile() it checks for full disk and allocates space prior to returning because the system cache holding the actual data is written later (a problem causes a delayed write error from the OS).
So the question is: would the same be true when using OVERLAPPED structure for asynchronous WriteFile() that expands the file beyond free space? e.g. It would return ERROR_DISK_FULL right away before pending the IO?
The reason to know is for recovery of freeing disk space, or inserting new media, and resuming the writes. If done this way, it's fairly straight forward, if after pending the IO, you could have a bunch of queued IO that then has to be synchronized and additional information tracked for all queued items in case moving to new media to adjust the offsets and such.
TIA!!
What you mean by asynchronous file operations (WriteFile() etc.) - these operations are only asynchronous for the caller. Internally they work the same way as synchronous (blocking) ones. The implementation of a blocking call invokes the non-blocking one and waits for an event the same as if you were using the OVERLAPPED structure. So, on your question of whether WriteFile would return ERROR_DISK_FULL before pending the IO, the answer is No. The rationale of non-blocking calls is not to make disk operation return results faster, but to allow a single thread to do multiple I/O operations in parallel without the need to create multiple threads.
if no enough disk space for complete write operation - you got ERROR_DISK_FULL (STATUS_DISK_FULL) when I/O operation will complete. are filesystem driver just complete your write request with STATUS_DISK_FULL (converted to ERROR_DISK_FULL) or first return STATUS_PENDING (converted to ERROR_IO_PENDING by win32) and then complete I/O with STATUS_DISK_FULL - this is undefined. can be both. final status will be ERROR_DISK_FULL but you cannot assume are operation will complete synchronous or asynchronous
I'm developing a charachter device driver for Linux.
I want to implement file-descriptor-targeted read() operation which will be a bit specific every time you open a device.
It is possible to identify the process where read() called from (using kernel current macro), but there can be several file descriptor associated with my device in this process.
I know that file descriptors got mapped to struct file objects just before making system call but can I get it back?
welcome to stackoverflow!
To achieve the goal you have specified in comment there are two methods:
ioctl and read :
Here you will have multiple buffers for each consumer to read from, and write buffer is different from read buffer. Each consumer immediatly after opening the device will fire an ioctl which will result in new buffer being allocated and a new token being generated for that buffer (something like this token numeber means this buffer). this token number should be passed back to the concernted consumer.
Now each consumer before making a read call will fire the ioctl giving the token number that will switch the current read buffer to that associated with that token number.
Now this method adds over head and you need to add locks too. Also no more than one consumer at a time can read from the device.
ioctl and mmap:
you can mmap the read buffer for each consumer and let it read from it at its own pace, using ioctl to request new data etc.
This will allow multiple consumers to read at the same time.
Or, you can malloc a new data buffer to read from on each open call and store the pointer to buffer in the private field of the file structure.
when ever a read is called this way you can just read the private data field of the file structure passed with the call and see which buffer is being talked about.
Also you can embed the whole structure containing the buffer pointer and size etc in the private field.
Short question, didn't seem to find anything useful here or on Google: in the Winsock2 API, is it possible to put data back in the sockets internal buffer when you have retrieved it using recv() for example, so that is seems it was never actually read from the buffer?
No, it is not possible to inject data back into the socket's internal buffer. Either use the MSG_PEEK flag to read data without removing it from the socket's buffer, or else read the socket data into your own buffer, and then do whatever you want with your buffer. You could have your reading I/O logic always look for data in your buffer first, and then read more data from the socket only when your buffer does not have enough data to satisfy the read operation. Any data you inject back into your buffer will be seen by subsequent read operations.
You can use the MSG_PEEK flag in your recv() call
Using libusb under Windows (where poll/select support is not quite there), is there a way to check for pending bytes on some USB endpoint, i.e. to check whether a following read will succeed (and read one or more bytes)?
I worked around the problem by implementing a ring buffer with already read bytes; checking for pending bytes looks in the ring buffer and (if it is empty) performs a non-blocking read on the endpoint (writing into the ring buffer if there's anything), and reading must use the ring buffer indirection instead of reading directly from the endpoint. This works for me, but is not very elegant.
I set my report size to 64 bytes and want to stream single reports (say 2 for now) to the host. My understanding is that there is a ReadFile buffer where these reports can sit. At the host, I have a 64 byte buffer that I use to read single reports. If I send one report from the device, the host reads it fine. If I use two ReadFiles in a loop, the second ReadFile times out. The device is sending two reports. I don't know if they're getting on the ReadFile buffer at the same time, so when the host reads the end point for the first report, the buffer gets purged and I lose the second report? If there are indeed 2 reports on the ReadFile buffer, do I need to read them both at once? How would I know how many reports are on the buffer?
ReadFile reads as many reports as there are in the HID driver's ring buffer up to the numberOfBytesToRead parameter.
The respective HID driver will implement everything as needed. You need not worry about whether those packets arrive "simultaneously". They won't.
The first packet should tell you the length of the report (i.e. a collection of packets), which in turn should allow you to figure out whether you have the full report, yet.
Of course you will have to keep an internal representation of the data from the report, because the packet buffers can only be at most 64 byte in size according to the specification. So to collect a full report you will have to handle that yourself or use the Hid_* routines described in the WDK.