I call CreateFile with FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH | FILE_FLAG_OVERLAPPED and then call many WriteFile with OVERLAPPED structures with both Offset and OffsetHigh members set to 0xFFFFFFFF to append new data to file.
Is it guaranteed that operations will be completed in same order as requested?
It seem logical for me but I see no explicit and non-ambiguous proofs of that.
Quote from https://support.microsoft.com/en-us/kb/156932 tells that operation is going to be synchronous:
On Windows NT, any write operation to a file that extends its length will be synchronous.
Great. Synchronous operation preserves order. But then:
The FILE_FLAG_NO_BUFFERING flag has the most effect on the behavior of the file system for asynchronous operation. This is the best way to guarantee that I/O requests are actually asynchronous.
Latter raised doubts in me. Could someone clarify this please?
Related
Wonder if anyone knows the internal design of WriteFile() (Storage Team Here?) with overlapped IO for file on on a disk drive/file system. Clearly when using the system buffer and standard synchronous WriteFile() it checks for full disk and allocates space prior to returning because the system cache holding the actual data is written later (a problem causes a delayed write error from the OS).
So the question is: would the same be true when using OVERLAPPED structure for asynchronous WriteFile() that expands the file beyond free space? e.g. It would return ERROR_DISK_FULL right away before pending the IO?
The reason to know is for recovery of freeing disk space, or inserting new media, and resuming the writes. If done this way, it's fairly straight forward, if after pending the IO, you could have a bunch of queued IO that then has to be synchronized and additional information tracked for all queued items in case moving to new media to adjust the offsets and such.
TIA!!
What you mean by asynchronous file operations (WriteFile() etc.) - these operations are only asynchronous for the caller. Internally they work the same way as synchronous (blocking) ones. The implementation of a blocking call invokes the non-blocking one and waits for an event the same as if you were using the OVERLAPPED structure. So, on your question of whether WriteFile would return ERROR_DISK_FULL before pending the IO, the answer is No. The rationale of non-blocking calls is not to make disk operation return results faster, but to allow a single thread to do multiple I/O operations in parallel without the need to create multiple threads.
if no enough disk space for complete write operation - you got ERROR_DISK_FULL (STATUS_DISK_FULL) when I/O operation will complete. are filesystem driver just complete your write request with STATUS_DISK_FULL (converted to ERROR_DISK_FULL) or first return STATUS_PENDING (converted to ERROR_IO_PENDING by win32) and then complete I/O with STATUS_DISK_FULL - this is undefined. can be both. final status will be ERROR_DISK_FULL but you cannot assume are operation will complete synchronous or asynchronous
It is very easy to determine if a message has been entirely read when reading form a pipe with read mode set to PIPE_READMODE_MESSAGE in synchronous I/O. (If the ReadFile function returns FALSE and GetLastError() returns ERROR_MORE_DATA, it means that the message is incomplete and that subsequent reads are necessary to retrieve the full message.)
Now, if the Named Pipe operates in Overlapped I/O instead and a read operation is pending (ReadFile function returns FALSE and GetLastError() returns ERROR_IO_PENDING), how do I know if I retrieved the full message when the operation completes? All I can determine is the number of bytes that were actually transferred by calling the GetOverlappedResult function, but it does not tell me whether or not the full message has been read…
Am I missing something here?
I think the easiest way ist to know, how long the messages is that you are expecting. Your protocol may give you the information.
For example the protocol always delivers a WORD as the first 2 bytes that tells you the length of the complete message.
So I use overlapped I/O with ReadFile to get the first 2 Bytes of the WORD. When I receive them I use ReadFile without overlapped I/O using the known message length and so I get all data.
What calls best emulate pread/pwrite in MSVC 10?
At the C runtime library level, look at fread, fwrite and fseek.
At the Win32 API level, have a look at ReadFile, WriteFile, and SetFilePointer. MSDN has extensive coverage of file I/O API's.
Note that both ReadFile and WriteFile take an OVERLAPPED struct argument, which lets you specify a file offset. The offset is respected for all files that support byte offsets, even when opened for synchronous (i.e. non 'overlapped') I/O.
Depending on the problem you are trying to solve, file mapping may be a better design choice.
It looks like you just use the lpOverlapped parameter to ReadFile/WriteFile to pass a pointer to an OVERLAPPED structure with the offset specified in Offset and OffsetHigh.
(Note: You don't actually get overlapping IO unless the handle was opened with FILE_FLAG_OVERLAPPED.)
The answer provided by Oren seems correct but doesn't seem to meet the needs. Actually, I too was here for searching the answer but couldn't find it. So, I will update a bit here.
As said,
At the C runtime library level, there are fread, fwrite and fseek.
At the Win32 API level, we can have two level of abstractions. One at the lower level which works with file descriptors and other at higher level which works with Windows' defined data structures such as File and Handle.
If you wish to work with Files and Handles, you have ReadFile, WriteFile, and SetFilePointer. But most the time, C++ developers prefer working with File Descriptors. For that, you have _read, _write and _lseek.
What happens if I ReadFile() 10 bytes (in overlapped mode without a timeout) but invoke CancelIo() after 5 bytes have been read? The documentation for CancelIo() says that it cancels any pending I/O, but what happens to the 5 bytes already read? Are they lost? Are they re-enqueued so the next time I ReadFile() I'll get them again?
I'm looking for the specification to indicate one way or another. I don't want to rely on empirical evidence.
According to http://groups.google.ca/group/microsoft.public.win32.programmer.kernel/browse_thread/thread/4fded0ac7e4ecfb4?hl=en
It depends on how the driver writer implemented the device. The exact
semantics of cancel on an operation are not defined to that level.
Either it doesn't matter because you are using overlapped I/O or you can just call SetFilePointer manually when you know you've cancelled I/O.
You don't have to rely on undocumented behavior if you just force the issue.
I understand the the ReadDirectoryChangesW Function uses a buffer to store the notifications, but what does overlap mean?
I presume there is protection to stop the notification you're reading from being over-written?
It is for asynchronous operations.
A call to ReadDirectoryChangesW can be completed synchronously or asynchronously. To specify asynchronous completion, open the directory with CreateFile as shown above, but additionally specify the FILE_FLAG_OVERLAPPED attribute in the dwFlagsAndAttributes parameter. Then specify an OVERLAPPED structure when you call ReadDirectoryChangesW.
See ReadDirectoryChangesW Function remarks sections.
this argument is for asynchronous operation.
on Windows, this is called "overlapped i/o". you can find this kind of parameter, with the same way of working, on a lot of function calls related to input/output (ReadFile, WriteFile, ...). more information about overlapped i/o can be found in the MSDN.