in one process, i called createfile with GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, OPEN_ALWAYS, and FILE_ATTRIBUTE_NORMAL as the params.
then i called LockFileEx on the whole file. i acquired an exclusive lock, and locked from range 0 to UINT_MAX.
after that, in another process, i called ::CreateFileW(path.c_str(), perms, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
and it erased the contents of the file. shouldn't it not be able to do that while my other prcoess has the file locked (exclusively)?
i want other processes to be able to get handles to the file (which is why i used the file_share flags), but i assumed they couldnt change the file at all while another process locked it, and so i was doing something like
createfile
lockfileex
and having it block on lockfileex until the other file releases the lock. is it just me or is msft's behavior here wrong?
It is debatable whether Windows should honor file locks when you attempt to open a file with CREATE_ALWAYS, but it appears from your notes that it does not. The best way to prevent other processes from writing to a file that you have open is not to specify FILE_SHARE_WRITE in your exclusive process.
However, this won't give you the waiting behavior that you would get from the file lock implementation without polling. There is no way to open a file with exclusive access and have any other files that are attempting to open it wait until the process with exclusive access relinquishes that access.
If you have access to the source of all of the processes involved, then you could just have the processes that are attempting to truncate the file first call LockFileEx on a region, and then call SetFileSize after acquiring the lock.
As a side note, only locking the region from 0 bytes to UINT_MAX won't provide mutual exclusion if, say, a process opens the file and attempts to write to the location at UINT_MAX + 1 (just past the 4GB point).
Related
How do i write to a physical drive in Windows 7?
I am trying to write to a physical disk (e.g. \\.\PhysicalDrive0) in Windows 7.
This question has been asked to death, but has never been answered. It is something that used to work in Windows XP, but Microsoft intentionally broke in Windows Vista. Microsoft provides hints about how to do it, but nobody has even been able to figure it out.
It used to work
In the olden days, writing to a physical disk was allowed (as long as you were an administrator). The method to do it was even documented in a Knowledge Base article:
INFO: Direct Drive Access Under Win32
To open a physical hard drive for direct disk access (raw I/O) in a Win32-based application, use a device name of the form
\\.\PhysicalDriveN
where N is 0, 1, 2, and so forth, representing each of the physical drives in the system.
You can open a physical or logical drive using the CreateFile() application programming interface (API) with these device names provided that you have the appropriate access rights to the drive (that is, you must be an administrator). You must use both the CreateFile() FILE_SHARE_READ and FILE_SHARE_WRITE flags to gain access to the drive.
All that changed in Windows Vista, when addition security restrictions were put in place.
How do you write to a physical disk?
Many people, and many answers, on many stackoverflow questions confuse:
writing to a physical disk (e.g. \\.\PhysicalDrive0), and
writing to a logical volume (e.g. \\.\C$)
Microsoft notes the restrictions placed on both kinds of operations:
Blocking Direct Write Operations to Volumes and Disks
Write operations on a DASD (Direct access storage device) volume handle will succeed if:
the file system is not mounted, or if
The sectors being written to are the boot sectors.
The sectors being written to reside outside file system space.
The file system has been locked implicitly by requesting exclusive write access.
The file system has been locked explicitly by sending down a lock/dismount request.
The write request has been flagged by a kernel-mode driver that indicates that this check should be bypassed. The flag is called SL_FORCE_DIRECT_WRITE and it is in the IrpSp->flags field. This flag is checked by both the file system and storage drivers.
In my case i am asking about writing to a Physical, not a Logical one. Microsoft notes the new set of restrictions on writing to a physical disk handle:
Write operations on a disk handle will succeed if:
The sectors being written to do not fall within a file system.
The sectors being written to fall within a mounted file system that is locked explicitly.
The sectors being written to fall within a file system that is not mounted or the volume has no file system.
My sectors being written do fall within a file system --> fail
My sectors being written do fall within mounted, unlocked, file system --> fail
My sectors being written do fall within a file system that is mounted, and in inside a logical volume that has a file system.
The hints on how to make it work revolve around:
unmounting a file system
locking a file system
But the question is how do you unmount a file system? How do you lock a file system?
What are you doing now?
I am able to read all physical sectors of a disk; that is no problem. The problem is when i want to write to a physical sector of the disk.
The current code i have is, in pseudo-code:
void ZeroSector(Int64 PhysicalSectorNumber)
{
String diskName := '\\.\PhysicalDrive0';
DWORD desiredAccess := GENERIC_READ or GENERIC_WRITE;
//INFO: Direct Drive Access Under Win32
//https://support.microsoft.com/en-us/kb/100027
//says you nedd both
DWORD shareMode := FILE_SHARE_READ or FILE_SHARE_WRITE;
//Open the physical disk
hDisk := CreateFile(diskName, desiredAccess, shareMode,
nil, OPEN_EXISTING, 0, 0);
if hDisk = INVALID_HANDLE_VALUE
RaiseLastWin32Error();
try
{
Int32 bytesPerPhysicalSector := 4096; //Determined elsewhere using IOCTL_STORAGE_QUERY_PROPERTY+STORAGE_ACCESS_ALIGNMENT_DESCRIPTOR
//Setup buffer for what we will be writing
Byte[] buffer = new Byte[bytesPerPhysicalSector];
//Calculate the byte offset of where the sector is
Int64 byteOffset = PhysicalSectorNumber * bytesPerPhysicalSector;
//Seek to that byte offset
SetFilePointer(hDisk, byteOffset.Lo, byteOffset.Hi, FILE_BEGIN);
//Write the buffer
DWORD numberOfBytesWritten;
if (!WriteFile(hDisk, buffer, bytesPerPhysicalSector, out numberOfBytesWritten, nil))
RaiseLastWin32Error();
}
finally
{
CloseHandle(hDisk);
}
}
Surprisingly:
i can open the physical disk for GENERIC_READ + GENERIC_WRITE access
it doesn't fail until the actual WriteFile, which fails with:
ERROR_ACCESS_DENIED
How to do what Microsoft says
Microsoft said that my write would fail, and they were right. They said that i need to explicitly lock the file system:
Write operations on a disk handle will succeed if:
The sectors being written to fall within a mounted file system that is locked explicitly.
Except i don't know how to do that.
I know i probably have to use DeviceIoControl and one of the IOCTLS to "lock" a volume. But that presents three challenges:
figuring out which volume(s) are on the physical disk selected
figuring out which IOCTL to use
figuring out how to unlock the locked volumes
Ignoring those problems, i blindly tried the LockFile API. Just before calling WriteFile:
//Try to lock the physical sector we will be writing
if (!LockFile(DiskHandle, byteOffset.Lo, byteOffset.Hi, bytesPerPhysicalSector, 0)
RaiseLastWin32Error();
That fails with:
ERROR_INVALID_FUNCTION (1)
Check out FSCTL_LOCK_VOLUME, FSCTL_DISMOUNT_VOLUME control codes. I believe you would have to enum all volumes you have on your disk, and then dismount-and-lock them. After lock succeeded, the disk is all yours.
You probably won't be able to do this on a system drive though. I'd also guess that there will be caveats with volumes that contain page files.
after I have done this, I can write to the corresponding \\.\PhysicalDrive3. I could not before:
HANDLE hVol = CreateFileA(
"\\\\.\\X:",
FILE_READ_DATA | FILE_WRITE_DATA,
FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
DWORD unused;
BOOL b = DeviceIoControl(hVol, FSCTL_DISMOUNT_VOLUME, NULL, 0, NULL, 0, &unused, NULL);
if (!b) {
printf("%u", GetLastError());
abort();
}
...
HANDLE h = CreateFileA(
argv[1], // that's my \\physicaldrive3
FILE_READ_DATA | FILE_WRITE_DATA,
FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
OPEN_EXISTING,
0,
NULL);
...
bResult = WriteFile(h, buf, cb, &dwWritten, &foo);
if (!bResult) {
// used to fail without messing with vol handle
printf("Failed writing data. Error = %d.\n", GetLastError());
return 0;
}
Editor FooEdit (let's call it) uses ReplaceFile() when saving to ensure that the save operation is effectively atomic, and that if anything goes wrong then the original file on disc is preserved. (The other important benefit of ReplaceFile() is continuity of file identity - creation date and other metadata.)
FooEdit also keeps open a handle to the file with a sharing mode of just FILE_SHARE_READ, so that other processes can open the file but can't write to it while it while FooEdit has it open for writing.
"Obviously", this handle has to be closed briefly while the ReplaceFile operation takes place, and this allows a race in which another process can potentially open the file with write access before FooEdit re-establishes it's FILE_SHARE_READ lock handle.
(If FooEdit doesn't close its FILE_SHARE_READ handle before calling ReplaceFile(), then ReplaceFile() fails with a sharing violation.)
I'd like to know what is the simplest way to resolve this race. The options seem to be either to find another way to lock the file that is compatible with ReplaceFile() (I don't see how this is possible) or to replicate all the behaviour of ReplaceFile(), but using an existing file handle to access the destination file rather than a path. I'm a bit stuck on how all of the operations of ReplaceFile() could be carried out atomically from user code (and reimplementing ReplaceFile() seems a bad idea anyway).
This must be a common problem, so probably there's an obvious solution that I've missed.
(This question seems related but has no answer: Transactionally write a file change on Windows.)
Here's a minimal verifiable example showing what I am trying to achieve (updated 13:18 30/9/2015 UTC). You must supply three file names as command line arguments, all on the same volume. The first must already exist.
I always get a sharing violation from ReplaceFile().
#include <Windows.h>
#include <stdio.h>
#include <assert.h>
int main(int argc, char *argv[])
{
HANDLE lock;
HANDLE temp;
DWORD bytes;
if (argc != 4)
{
puts("First argument is the project file. Second argument is the temporary file.");
puts("The third argument is the backup file.");
}
/* Open and lock the project file to make sure no one else can modify it */
lock = CreateFile(argv[1], GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_DELETE, NULL, OPEN_EXISTING, 0, 0);
assert(lock != INVALID_HANDLE_VALUE);
/* Save to the temporary file. */
temp = CreateFile(argv[2], GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_DELETE, NULL, CREATE_ALWAYS, 0, 0);
assert(temp != INVALID_HANDLE_VALUE);
WriteFile(temp, "test", 4, &bytes, NULL);
/* Keep temp open so that another process can't modify the file. */
if (!ReplaceFile(argv[1], argv[2], argv[3], 0, NULL, NULL))
{
if (GetLastError() == ERROR_SHARING_VIOLATION)
puts("Sharing violation as I expected");
else
puts("Something went wrong");
}
else
puts("ReplaceFile worked - not what I expected");
/* If it worked the file referenced by temp would now be called argv[1]. */
CloseHandle(lock);
lock = temp;
return EXIT_SUCCESS;
}
Thanks to Hans Passant, who provided some valuable clarifying thoughts in an answer now deleted. Here's what I discovered while following up his suggestions:
It seems ReplaceFile() allows lpReplacedFileName to be open FILE_SHARE_READ | FILE_SHARE_DELETE, but lpReplacementFileName can't be. (And this behaviour doesn't seem to depend on whether lpBackupFileName is supplied.) So it's perfectly possible to replace a file that another process has open even if that other process doesn't allow FILE_SHARE_WRITE, which was Hans' point.
But FooEdit is trying to ensure no other process can open the file with GENERIC_WRITE in the first place. To ensure in FooEdit that there's no race where another process can open the replacement file with GENERIC_WRITE, it seems that FooEdit has to keep hold continuously of a FILE_SHARE_READ | FILE_SHARE_DELETE handle to lpReplacementFileName, which then precludes use of ReplaceFile().
Actually I think there might be a solution that doesn't involve transactions (although transactions are still available as far as I know). I haven't tried it myself, but I think on NTFS it should be possible to create a new file stream (use a long random name to ensure there are no collisions), write your data, and then rename that stream to the stream you actually wanted to write to.
FILE_RENAME_INFORMATION suggests this should be possible, since it talks about renaming data streams.
However, this would only work on NTFS. For other file systems I don't think you have a choice.
I'd like to know what is the simplest way to resolve this race.
There is no simple way to resolve this race. It's an inherent part of the file system which is not transactional. MS introduced a transactional file API with Vista but now strongly advise developers not to use it as it may be removed in a future release.
I have had some experience with ReplaceFile but I think it caused more trouble than it was worth. My recollection was that whilst meta data was preserved, a new file was created. A consequence of this was very annoying behaviour for files saved on the desktop. Because such files have their position preserved, creating a new file resulted in the default position being used. So you'd save a file, you'd drag it to the place on the desktop where you wanted to keep it, and then when you saved the file again, it moved back to the default position.
I wrote a very simple code to set up a "server" that creates a named pipe and waits for a client to connect. As soon as the client opens the pipe, the server sends its data (a block of about 10mb) and the client is supposed to read it and close the connection.
The real catch now is: When the pipe is working with local names (\.\pipe\xxx) it does send all the data without any problem but if i change the path to a network name (\computer\pipe\xxx) it changes behavior and client can only read about 65000~ bytes, but it does not complete read operation even when i loop it (i suppose it breaks in 65k blocks to send over network since i'm using a network name, it happens even locally). ReadFile reads the 65k block and returns TRUE, if i try to force ReadFile again in the pipe it reads 0 bytes.
The flags i'm using to create the pipe are PIPE_ACCESS_DUPLEX, FILE_FLAG_WRITE_THROUGH, PIPE_TYPE_BYTE, PIPE_READMODE_BYTE, PIPE_WAIT, PIPE_ACCEPT_REMOTE_CLIENTS
Here is a piece of what the code should look like (the code is somewhere else and i can't access it right now but as i said before, it is as simple as it gets)
lPipe := CreateFileA('\\.\pipe\test', GENERIC_READ or GENERIC_WRITE, FILE_SHARE_READ or FILE_SHARE_WRITE, nil, OPEN_EXISTING, 0, 0);
ReadFile(lPipe, lMemoryStream.Memory^, 1024*1024*15, lBytesRead, nil);
CloseHandle(lPipe);
From the MSDN documentation for WriteFileEx:
Pipe write operations across a network are limited to 65,535 bytes per write. For more information regarding pipes, see the Remarks section.
To get past this, you'll have to set up the server to send the data in chunks.
I'm trying to use FILE_FLAG_BACKUP_SEMANTICS to bypass file permissions, but it doesn't work if I request GENERIC_ALL or the equivalent FILE_ALL_ACCESS as the access mask. The CreateFile function returns ERROR_ACCESS_DENIED.
For example,
h = CreateFile(L"c:\\working\\backup-semantics-test\\xyzzy\\test.txt",
FILE_ALL_ACCESS,
FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE, 0,
OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, 0);
Backup and restore permissions are enabled. (Obviously, it works if I have full permissions to the file in question; this question is about the case where I do not have permissions but do have backup and restore privilege.)
What's going wrong, and what can I do instead?
You cannot obtain FILE_DELETE_CHILD access via FILE_FLAG_BACKUP_SEMANTICS. This appears to be a special case, although it is not obvious whether this is deliberate or not. (Tested in Windows 7 SP1 x64.)
You can obtain all other access rights via backup semantics, including DELETE, so obtaining FILE_DELETE_CHILD is unnecessary. Simply exclude it from the request:
h = CreateFile(L"c:\\working\\backup-semantics-test\\xyzzy\\test.txt",
FILE_ALL_ACCESS & ~FILE_DELETE_CHILD,
FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE, 0,
OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, 0);
If possible, request only those permissions that you actually need. In my case, I was intending to call SetFileShortName, so I set the access mask to GENERIC_WRITE|DELETE as documented.
Another option is MAXIMUM_ALLOWED but I don't recommend using this option, except during troubleshooting, because it means that you might not get the access rights that you're expecting. It is usually best to discover the problem as soon as you open the file rather than later on when you're trying to do something with it.
Starting with the ndisprot sample from Microsoft I try to write a NDIS protocol driver. From User space I try to read and write to the device simultaneous (out of two threads). Since I don't receive any packets, the ReadFile system call blocks. I'm not able to complete a WriteFile system call in this state.
CHAR NdisProtDevice[] = "\\\\.\\\\NDISprot";
CHAR * pNdisProtDevice = &NdisProtDevice[0];
this.iHandle = CreateFile(pNdisProtDevice,
GENERIC_WRITE | GENERIC_READ, 0, 0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);
// Blocks, because no frames arrive
bSuccess = (BOOLEAN)ReadFile(Handle,
(LPVOID)pReadBuf,
PacketLength,
&BytesRead,
NULL);
...
// Called some seconds later from another thread, while ReadFile still blocking...
bSuccess = (BOOLEAN)WriteFile(Handle,
pWriteBuf,
PacketLength,
&BytesWritten,
NULL);
I added some debug messages and discovered that the driver function associated with IRP_MJ_WRITE (NdisprotWrite) gets not even called! Something between the user space application and the driver blocks concurrent access to the device \Device\NDISprot.
How can I concurrent Read and Write to the file?
By default, you can only have one outstanding I/O request per usermode handle. Either open multiple handles, or open your one handle with FILE_FLAG_OVERLAPPED. (Once you use FILE_FLAG_OVERLAPPED, you also generally need to use OVERLAPPED structures - make sure you've got the gist of it by skimming this and this.)