Supposedly it is possible to actually open and read directories on NTFS volumes. However, my code to try this wasn't working, so I tried google, which found me this.
The key observation there seems to be that you must use FILE_FLAG_BACKUP_SEMANTICS. So, trimming that down, I basically get:
HANDLE hFile = CreateFile(L"C:\\temp", GENERIC_READ, FILE_SHARE_READ,
0, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, 0);
DWORD dwFileSize = GetFileSize(hFile, 0);
char* buf = new char[dwFileSize];
DWORD dwBytesRead = 0;
BOOL b = ReadFile(hFile, buf, dwFileSize, &dwBytesRead, 0);
Seems pretty straight-forward. Unfortunately, it doesn't work.
The CreateFile and GetFileSize both work (handle is not INVALID_HANDLE_VALUE, non-zero and plausible file size), but the ReadFile returns FALSE, dwBytesRead is zero, and GetLastError returns 1 ("Incorrect function"). Huh.
While I was typing this question, the 'Similar Questions' prompt showed me this. That business about using AdjustTokenPrivileges made a lot of sense. However, it didn't help. Adding ReadFile (and using c:\temp) to that example gives the same behavior. A closer reading of the CreateFile docs shows that even without the SE_BACKUP_NAME privilege, I should be able to open the file due to admin privileges.
I've tried a number of permutations:
Different ways of specifying the directory name (c:\temp, c:\temp\, \\.\c:\temp, \\?\c:\temp\, etc).
Different directories
Different drives
Different share options (0, FILE_SHARE_READ, FILE_SHARE_READ | FILE_SHARE_WRITE)
Different access permissions (GENERIC_READ, FILE_LIST_DIRECTORY, FILE_LIST_DIRECTORY + FILE_READ_EA + FILE_READ_ATTRIBUTES, FILE_LIST_DIRECTORY + FILE_READ_EA + FILE_READ_ATTRIBUTES + FILE_TRAVERSE)
I can't see any flags that might apply other than FILE_FLAG_BACKUP_SEMANTICS (which I assume is required), but I tried FILE_FLAG_NO_BUFFERING and a 4096 byte aligned buffer. Nope.
I'm (currently) trying 152 permutations, and none of the ReadFiles are working. What am I missing?
Is my original assumption here incorrect? Is it not really possible to 'read' from a directory? Or is there just some trick I'm still missing?
What else should I mention?
I'm running as an admin, and can do a CreateFile on the volume.
My program is 64bit, built for unicode.
Windows 7 x64
NTFS 3.1 volume
It's cloudy outside (Hey, you never know what might matter...)
If you want to open a stream then you need to include the stream name and/or type as part of the path:
c:\foo:bar A.K.A. c:\foo:bar:$DATA
c:\foo::$INDEX_ALLOCATION
The default $DATA stream is used if you don't specify a stream. $DATA stores a files "normal data".
If you want the list of files in the directory then you can use GetFileInformationByHandleEx(FileIdBothDirectoryInfo) (and NtQueryDirectoryFile on older systems).
It looks like Jonathan Potter has given the correct answer. Despite prompting, he has elected not to post his comments as an answer. So I'm going to create one based on his responses in order to close the question.
In short: "You can open a handle to a directory to do certain things, but calling ReadFile on it isn't one of them."
What things? These things. This list includes:
BackupRead
BackupSeek
BackupWrite
GetFileInformationByHandle
GetFileSize
GetFileTime
GetFileType
ReadDirectoryChangesW
SetFileTime
In summary: While you can "open" directories and "read" certain information about them, you can't actually use ReadFile. If you want to read the DirName::$INDEX_ALLOCATION information, you'll have to use a different approach.
Related
Editor FooEdit (let's call it) uses ReplaceFile() when saving to ensure that the save operation is effectively atomic, and that if anything goes wrong then the original file on disc is preserved. (The other important benefit of ReplaceFile() is continuity of file identity - creation date and other metadata.)
FooEdit also keeps open a handle to the file with a sharing mode of just FILE_SHARE_READ, so that other processes can open the file but can't write to it while it while FooEdit has it open for writing.
"Obviously", this handle has to be closed briefly while the ReplaceFile operation takes place, and this allows a race in which another process can potentially open the file with write access before FooEdit re-establishes it's FILE_SHARE_READ lock handle.
(If FooEdit doesn't close its FILE_SHARE_READ handle before calling ReplaceFile(), then ReplaceFile() fails with a sharing violation.)
I'd like to know what is the simplest way to resolve this race. The options seem to be either to find another way to lock the file that is compatible with ReplaceFile() (I don't see how this is possible) or to replicate all the behaviour of ReplaceFile(), but using an existing file handle to access the destination file rather than a path. I'm a bit stuck on how all of the operations of ReplaceFile() could be carried out atomically from user code (and reimplementing ReplaceFile() seems a bad idea anyway).
This must be a common problem, so probably there's an obvious solution that I've missed.
(This question seems related but has no answer: Transactionally write a file change on Windows.)
Here's a minimal verifiable example showing what I am trying to achieve (updated 13:18 30/9/2015 UTC). You must supply three file names as command line arguments, all on the same volume. The first must already exist.
I always get a sharing violation from ReplaceFile().
#include <Windows.h>
#include <stdio.h>
#include <assert.h>
int main(int argc, char *argv[])
{
HANDLE lock;
HANDLE temp;
DWORD bytes;
if (argc != 4)
{
puts("First argument is the project file. Second argument is the temporary file.");
puts("The third argument is the backup file.");
}
/* Open and lock the project file to make sure no one else can modify it */
lock = CreateFile(argv[1], GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_DELETE, NULL, OPEN_EXISTING, 0, 0);
assert(lock != INVALID_HANDLE_VALUE);
/* Save to the temporary file. */
temp = CreateFile(argv[2], GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_DELETE, NULL, CREATE_ALWAYS, 0, 0);
assert(temp != INVALID_HANDLE_VALUE);
WriteFile(temp, "test", 4, &bytes, NULL);
/* Keep temp open so that another process can't modify the file. */
if (!ReplaceFile(argv[1], argv[2], argv[3], 0, NULL, NULL))
{
if (GetLastError() == ERROR_SHARING_VIOLATION)
puts("Sharing violation as I expected");
else
puts("Something went wrong");
}
else
puts("ReplaceFile worked - not what I expected");
/* If it worked the file referenced by temp would now be called argv[1]. */
CloseHandle(lock);
lock = temp;
return EXIT_SUCCESS;
}
Thanks to Hans Passant, who provided some valuable clarifying thoughts in an answer now deleted. Here's what I discovered while following up his suggestions:
It seems ReplaceFile() allows lpReplacedFileName to be open FILE_SHARE_READ | FILE_SHARE_DELETE, but lpReplacementFileName can't be. (And this behaviour doesn't seem to depend on whether lpBackupFileName is supplied.) So it's perfectly possible to replace a file that another process has open even if that other process doesn't allow FILE_SHARE_WRITE, which was Hans' point.
But FooEdit is trying to ensure no other process can open the file with GENERIC_WRITE in the first place. To ensure in FooEdit that there's no race where another process can open the replacement file with GENERIC_WRITE, it seems that FooEdit has to keep hold continuously of a FILE_SHARE_READ | FILE_SHARE_DELETE handle to lpReplacementFileName, which then precludes use of ReplaceFile().
Actually I think there might be a solution that doesn't involve transactions (although transactions are still available as far as I know). I haven't tried it myself, but I think on NTFS it should be possible to create a new file stream (use a long random name to ensure there are no collisions), write your data, and then rename that stream to the stream you actually wanted to write to.
FILE_RENAME_INFORMATION suggests this should be possible, since it talks about renaming data streams.
However, this would only work on NTFS. For other file systems I don't think you have a choice.
I'd like to know what is the simplest way to resolve this race.
There is no simple way to resolve this race. It's an inherent part of the file system which is not transactional. MS introduced a transactional file API with Vista but now strongly advise developers not to use it as it may be removed in a future release.
I have had some experience with ReplaceFile but I think it caused more trouble than it was worth. My recollection was that whilst meta data was preserved, a new file was created. A consequence of this was very annoying behaviour for files saved on the desktop. Because such files have their position preserved, creating a new file resulted in the default position being used. So you'd save a file, you'd drag it to the place on the desktop where you wanted to keep it, and then when you saved the file again, it moved back to the default position.
I use Lazarus 1.2.2 and Freepascal 2.6.4.
I have a program called QuickHash that hashes files and, when I run it on Linux, it can be used to hash physcial disks too (/dev/sdXX). However, I'd like to add the ability to the Windows version.
I gather that to access physcial devices, like disks, one has to use CreateFile. Specifically, CreateFileW.
So, the user clicks a button which scans the computer for disks and lists them in a List Box. The one the user double clicks is then parsed (ListBox.GetSelectedText) for the string '\.\PhyscialDiskX' and that is assigned to a string variable,
strDiskID := getDiskID(Listbox.GetSelectedText);
That works fine.
I then try to create a handle to that disk :
hSelectedDisk := Windows.CreateFileW(PWideChar(strDiskID), GENERIC_READ, FILE_SHARE_READ, nil, OPEN_EXISTING, FILE_FLAG_RANDOM_ACCESS, 0);
Based on this article, specifcally "You must use both the CreateFile() FILE_SHARE_READ and FILE_SHARE_WRITE flags to gain access to the drive" I have also tried the other two combinations below :
hSelectedDisk := CreateFileW(PWideChar(strDiskID), GENERIC_READ, FILE_SHARE_READ AND FILE_SHARE_WRITE, nil, OPEN_EXISTING, FILE_FLAG_RANDOM_ACCESS, 0);
hSelectedDisk := CreateFileW(PWideChar(strDiskID), GENERIC_READ, FILE_SHARE_READ OR FILE_SHARE_WRITE, nil, OPEN_EXISTING, FILE_FLAG_RANDOM_ACCESS, 0);
All three successfully assign a handle. But, the top syntax and the bottom syntax eventually generate an error (explained below). The middle option immediately returns the default initialisation hash for a zero byte file, i.e. DA39... for SHA1.
My problem is I am unable to pass that handle (which is an integer) to the SHA1File and MD5FILE functions of the Freepascal md5 and SHA1 units. They expect a filename, which has to be a string.
So, if I pass it the strDiskID ('\.\PhyscialDiskX') (which defeats the objects of assigning a handle at all) I do get disk activity and the program appears to be working.
strDiskHashValue := SHA1Print(SHA1File(strDiskID));
ShowMessage(strDiskHashValue);
CloseHandle(hSelectedDisk);
But even when run on really small disks like a 500Mb USB drive, it takes many minutes and eventually returns "Run Error 1117" which according to this means
"ERROR_IO_DEVICE
1117 (0x45D)
The request could not be performed because of an I/O device error."
However, I have tried it on several working disks and the error continues.
So, my question, ultimately, is how on earth do I pass that successfully assigned THandle to the hashing functions? (UI have also asked the question at the Lazarus forums but sometimes I get answers here from members who don't see the threads there)
You aren't going to be able to pass volume handles to functions that aren't expecting volume handles. These are very special handles with quite stringent requirements on their use. Not the least of which is that you must read blocks that are sector aligned, and whose sizes are multiples of the sector size.
So the solution is for you to take charge of reading the data. Read it into a buffer, and then pass that buffer to the hashing library. This means that you'll need a hashing library that can be called repeatedly to process new data. All comprehensive hashing libraries will offer such functionality.
The share mode flags are combined with bitwise or:
FILE_SHARE_READ or FILE_SHARE_WRITE
I would create the handle like this:
hSelectedDisk := CreateFileW(PWideChar(strDiskID), FILE_READ_DATA,
FILE_SHARE_READ or FILE_SHARE_WRITE, nil, OPEN_EXISTING, 0, 0);
First of all I would concentrate on reading the content of the volume. Once you can do that, hashing will be routine.
It seems from the comments that you are having some trouble writing the hashing code. First of all you need to allocate a buffer that is a multiple of the sector size:
var
Buffer: Pointer;
....
GetMem(Buffer, BufferSize);
Use IOCTL_DISK_GET_DRIVE_GEOMETRY to find out the sector size. And take note of this text from the documentation:
To read or write to the last few sectors of the volume, you must call DeviceIoControl and specify FSCTL_ALLOW_EXTENDED_DASD_IO. This signals the file system driver not to perform any I/O boundary checks on partition read or write calls. Instead, boundary checks are performed by the device driver.
Now that you have a buffer, you can read and hash the content.
var
ctx: TSHA1Context;
Digest: TSHA1Digest;
BytesRead: DWORD;
....
SHA1Init(ctx);
repeat
if not ReadFile(hSelectedDisk, Buffer^, BufferSize, BytesRead, nil) then
// handle error, raise exception
SHA1Update(ctx, Buffer^, BytesRead);
until BytesRead < BufferSize;
SHA1Final(ctx, Digest);
I've not attempted to compile or test this code. It's not meant to be complete or comprehensive. It is just intended to show you how to tackle the problem.
I'd like to change a file when it is closed and reverse the change when it is opened.
It's kind of like encryption driver except I don't want to encrypt the file.
I've created a new "Filter Driver: Filesystem Mini-Filter" project with WDK8 in Visual Studio 2012 and registered PreCreate, PostCreate, PreClose and PostClose as callback functions.
For example, on IRP_MJ_CLOSE of file which it's byte are {72,101,108,108,111} ("Hello"), I want that after the PostClose function the file would look like this on the hard disk:
{10,11,12,72,101,108,108,111}.
I suspect it is not as easy as just:
FLT_PREOP_CALLBACK_STATUS
PreClose (
_Inout_ PFLT_CALLBACK_DATA Data,
_In_ PCFLT_RELATED_OBJECTS FltObjects,
_Flt_CompletionContext_Outptr_ PVOID *CompletionContext
)
{
//...
//some if statment...
{
Data->Iopb->Parameters.Write.WriteBuffer = newBfr;
Data->Iopb->Parameters.Write.Length = newLen;
}
//...
return FLT_PREOP_SUCCESS_WITH_CALLBACK;
}
I'd like some guidance on the subject.
Also what is the best way to debug this? I Haven't found a way to print to the windows 7 debug.
Thanks!
gfgqtmakia.
EDIT: I've read http://code.msdn.microsoft.com/windowshardware/swapBuffer-File-System-6b7e6e2d but I don't think it'll help me because it is for read/write, which I don't want to deal with.
EDIT2: Or maybe I should do my changes in the PreCreate and PostClose, when the file is on the hard drive and not in the middle of an IRP, and then I won't need to deal with buffers "on the fly" but on the disk?
You will have to write something like swap buffers. Modifying file data in PostCreate/PreClose would not be good idea.
Few reasons:
Firstly in PostCreate/PreClose you shouldn't be accessing Data->Iopb->Parameters.Write.WriteBuffer. That is valid only in IRP_MJ_WRITE. You can do FltWriteFile to write data to file.
Windows kernel may not write file data immediately to the disk in/after IRP_MJ_CLOSE. Think about page cache.
There are may complexities like paging i/o, direct i/o etc. that need to be taken care properly.
Another major thing I notice it that you will also change the file size (as said in your question actual data length is 5 bytes while you will update data to 8 bytes). Now this is very difficult to manage. It never recommended to change the file size in minifilter/file system driver.
I am currently writing a Windows utility that does a similar job as Linux dd. But this utility only copy USED block instead of cloning the whole disk block by block.
The way I write it is to
*
(1) Copy MBR from disk A which contains three partitions to disk B.
(2) after step A. I am able see three raw partitions from disk management GUI. (no drive letter, no file system).
(3) Copy each partitions from A to B in a loop.*
The problem I have is:
in step (3) after I get the file handle from OpenDevice and I try to use that handle to lock a raw partition, I get Access Denied (error 5) and I am 100% sure that my program did not accidentally hold the lock somewhere else. My dev and testing environment is Windows 2003 server and my utility is executed as Admin privilege.
Here is the code snippet:
...
shareMode = FILE_SHARE_READ | FILE_SHARE_WRITE;
fHandle= OpenDevice(shareMode);
res = DeviceIoControl(fHandle, FSCTL_LOCK_VOLUME, NULL, 0, NULL, 0, &dummy, NULL);
if (res == 0)
{
DismountAndLockVolume();
}
...
in my OpenDevice function in use:
DWORD access = (fOpenMode==forWriting) ? (GENERIC_READ | GENERIC_WRITE | SYNCHRONIZE) : GENERIC_READ| SYNCHRONIZE;
NTOpen(&fHandle, fName.c_str(), access, FILE_ATTRIBUTE_NORMAL, shareMode, FILE_OPEN,FILE_SYNCHRONOUS_IO_NONALERT|FILE_RANDOM_ACCESS|FILE_NON_DIRECTORY_FILE);
to get the file handle(fHandle).
In DismountAndLockVolume function, I try to dismount this volume and grab the lock on it again.
DeviceIoControl(fHandle, FSCTL_DISMOUNT_VOLUME, NULL, 0, NULL, 0, &dummy, NULL);
DWORD shareMode = FILE_SHARE_DELETE | FILE_SHARE_WRITE | FILE_SHARE_READ;
ntStatus = OpenDevice(shareMode);
res = DeviceIoControl(fHandle, FSCTL_LOCK_VOLUME, NULL, 0, NULL, 0, &dummy, NULL);
The problem is, when I my code try to dismount this volume, an exception is thrown due to access denied.
Then I browsed the Internet and get the following discussion thread: QUOTE
A write on a volume handle will succeed if the volume is not mounted by a
file system, or if one of the following conditions is true:
1. The sectors to be written to are boot sectors.
2. The sectors to be written to reside outside of file system space.
3. You have explicitly locked or dismounted the volume by using
FSCTL_LOCK_VOLUME or FSCTL_DISMOUNT_VOLUME.
4 . The volume has no file system. (In other words, it has been mounted
as a RAW volume.)
A write on a disk handle will succeed if one of the following conditions
is true:
1. The sectors to be written to do not fall within a volume's extents.
2. The sectors to be written to fall within a mounted volume, but you
have explicitly locked or dismounted the volume by using FSCTL_LOCK_VOLUME
or FSCTL_DISMOUNT_VOLUME.
3. The sectors to be written to fall within a volume that is not mounted
or has no file system.
The modification of some disk parts, like the boot sector ( upto 16 ), is
still allowed . But my utility relies on raw write access to the disk. I am
not able to lock volume by FSCTL_LOCK_VOLUME. Result is ACCESS DENIED. While
searching through net i come to know that kernel mode driver is only
solution. But in this group i come to know driver is not require. I work on
services and pass through SCSI, but i am not able to find the solution. If
kernel mode driver is require then what kind of driver it should be ?
Is anybody know the solution of the problem then please help me out
This thread terminates with no conclusion. Can somebody give me a hand on what was going on? Any hint for possible solution?
Millions of Thanks. I have been scratching my hair for days :-(((.
I'm trying to register a performance counter and part of this process includes adding some textual descriptions to a specific registry key. For English this key is HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib\009 which apparently is also known as HKEY_PERFORMANCE_TEXT. There are a pair of values under there (Counter, Help) that have REG_MULTI_SZ data, and I need to modify them to accomplish my goal.
The official way of doing this is by using a tool called lodctr along with a .h and .ini file. There is also a function for doing this programmatically, but my understanding is that it is just a simple wrapper around calling the lodctr program. I found the prospect of maintaining, distributing, and keeping synchronized 3 separate files a bit cumbersome, so I previously wrote code to do this and it worked fine under Windows XP (and possibly Vista, though I don't remember for sure).
Now I'm trying to use the same code on Windows 7 and it doesn't work. The problem is that whenever I try to set the registry values it fails with ERROR_BADKEY; even regedit fails to modify the values, so it's not a problem with my code. I ran Process Monitor against it and noticed that there was no activity at the driver level, so it seems this access must be getting blocked in user-mode code (e.g. advapi32.dll or wherever). I understand why Microsoft would try to prevent people from doing this as it is very easy to screw up, and doing so will screw up the entire performance counter collection on the machine.
I'm going to debug lodctr and see what the magic is purely out of curiosity, but I'm wondering if anybody has run into this before? Are there any alternatives other than the lodctr utility? Perhaps calling the NT registry API directly? I would really prefer to avoid the hassle of the lodctr method if possible.
A minimal example to reproduce the issue:
HKEY hKey = NULL;
LONG nResult = RegOpenKeyEx(HKEY_LOCAL_MACHINE, _T("SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Perflib\\009"), 0, KEY_ALL_ACCESS, &hKey);
if(ERROR_SUCCESS == nResult)
{
LPCTSTR lpData = _T("bar");
DWORD cbData = (_tcsclen(lpData) + 1) * sizeof(TCHAR);
nResult = RegSetValueEx(hKey, _T("foo"), 0, REG_SZ, (const BYTE*)lpData, cbData);
// here nResult == ERROR_BADKEY
RegCloseKey(hKey);
hKey = NULL;
}
EDIT 1:
I spent about an hour or so trying to debug the official APIs and couldn't figure it out so I tried some more Google. After a while I came across this KB article which explains the RegSetValueEx behavior. Since it mentioned modifying system files that got me to thinking that perhaps this particular registry data is backed by a mapped file. Then I came across another KB article that mentions Perfc009.dat and Perfh009.dat in the system32 folder. Opened these up in a hex editor and sure enough it is the raw REG_MULTI_SZ data I am trying to modify. Now that I know that maybe I can take another look and figure it out, though I am bored with it for now.
Never mind, I give up. It's easier to just go with the flow. Instead of trying to modify the registry directly, I will create the .h and .ini files programmatically and invoke the relevant functions.