In my program I call ReadDirectoryChangesW to listen for file events in a given directory. The problem is some events (e.g. FILE_ACTION_ADDED) are signaled when the file is opened, not closed. This means the file will be locked by other process for some unspecified amount of time and CreateFileW will be returning an error.
The question is: how do I open the file when the other process is done with it? I can tolerate race conditions (e.g. some other process manages to delete the file after it's closed, but before I open it), but I'd like to avoid busy waiting.
Options I see/considered so far:
Asynchronous CreateFileW. That would be an ideal solution, but it's not possible - all user-space APIs for opening a file are synchronous by design (see a great explanation).
Listening for FILE_NOTIFY_CHANGE_LAST_ACCESS. This almost works - notification on close is sent only when the other process wrote some bytes to the file.
I also found some resources on kernel filter drivers, which can detect file close event. Probably works, but seems a bit too complex.
Busy loop continuously calling CreateFileW until it succeeds. Overutilizes the CPU, but is the only thing that actually works. I'm worried I'm stuck with this approach.
Related
I'm using Transactional NTFS to atomize multiple writes to several files.
The problem is that after commit, I may not be able to reopen a file,
perhaps because of a racing condition.
The sequence of events is :
NTFS transaction is created with CreateTransaction
Files are opened with CreateFileTransacted
Writes are done to the files
Files are closed with CloseHandle
Transaction is committed with CommitTransaction
Files are reopened with CreateFile for read/write
The last step sometimes fails with error code 3 :
ERROR_PATH_NOT_FOUND - The system cannot find the path specified.
When re-executing the program, the file is then found.
This happens rarely, but in a completely random manner, meaning not always
when reopening the same file.
My theory is that if terminating the transaction by Windows takes a long
time, the files are not available for opening in read/write mode until
the transaction terminates. My program then fails when trying to open
my own files in non-transaction mode.
I think that to avoid this problem, I need to wait for the transaction
to complete before reopening the files.
However, I have not found any documented method for doing that.
No clever answers, so I had to implement my own dummy one :
If an I/O error happens on opening a file that was just closed,
the solution was to loop on opening several times while in-between calling
Sleep() to release the CPU, before deciding that a catastrophic error had occurred.
Dummy solution, but it solved the problem.
When is it necessary to flush a file?
I never do it because I call File.Close and I think that it is flushed automatically, isn't it?
You'll notice that an os.File doesn't have a .Flush() because it doesn't need one because it isn't buffered. Writes to it are direct syscalls to write to the file.
When your program exits(even if it crashes) all files it has open will be closed automatically by the operating system and the file system will write your changes to disk when it gets around to it (sometimes up to few minutes after your program exits).
Calling os.File.Sync() will call the fsync() syscall which will force the file system to flush it's buffers to disk. This will guarantee that your data is on disk and persistent even if the system is powered down or the operating system crashes.
You don't need to call .Sync()
See here. File.Sync() is syscall to fsync. You should be able to find more under that name.
Keep in mind that fsync is not the same as fflush and is not executed before close.
You generally don't need to call it. The file will be written to disk anyway, after some time and if there is no power failure during this period.
Looks the most recommendations here are to not to call fsync(), but in general it mainly depends on your application requirement. If you are working on critical file read/write, its always recommended to call fsync().
http://www.microhowto.info/howto/atomically_rewrite_the_content_of_a_file.html#idp31936
link, has more details on when file.Sync() will help.
When you want to ensure data integrity as much as possible. For example, what happens if your program crashes before it comes to closing the file?
I am working on an MFC application that can (among other things) be used to shut Windows down. When doing this, Windows of course sends the WM_QUERYENDSESSION and WM_ENDSESSION to all applications, mine included. However, the problem is that my application, as part of some destructors, delete certain files (with CFile::Remove) that have been used during the execution. I have reason to believe that the destructors are called (but that is hard to know for certain) when the application is closed by Windows.
However, when Windows starts back up again, I do occasionally notice that the files that were supposed to be deleted are still present. This does not happen consistently, even when the execution of the program is identical (I have a script for testing this). This leads me to think that one of two things are happening: Either a) the destructors are not consistently being called, or b) the Remove function returns, but the file is not actually deleted before Windows is shut down.
The only work-around I have found so far is that if I get the system to wait with the shutdown for approximately 10 seconds after my program has stopped, then the files will be properly deleted. This leads me to believe that b) may be the case.
I hope someone is able to help me with this problem.
Regards
Mort
Once your program returns from WM_ENDSESSION, Windows can terminate it at any time:
If the session is being ended, this parameter is TRUE; the session can end any time after all applications have returned from processing this message.
If the session ends quickly, then it may end before your destructors run. You must do all your cleanup before returning from WM_ENDSESSION, because there is no guarantee that you will get a chance to do it afterwards.
The problem here is that some versions of Windows report back that file handling operations have been completed before they actually have. This isn't a problem unless shutdown is triggered as some operations, including file delete will be abandoned.
I would suggest that you cope with this by forcing your code to wait for a confirmed deletion of the files (have a process look for the files and raise an event when they've gone) before calling for system shutdown.
If the system is properly shut down (nut went sudden power loss or etc.) then all the cached data is flushed. In particular this includes flushing the global file descriptor table (or whatever it's called in your file system) which should commit the file deletion.
So the problem seems to be that the user-mode code doesn't call DeleteFile, or it failes (for whatever reason).
Note that there are several ways the application (process) may exit, whereas not always d'tors are called. There are automatic objects which are destroyed in the context of their callstack, plus there are global/static objects, which are initialized and destroyed by the CRT init/cleanup code.
Below is a short summary of ways to terminate the process, with the consequences:
All process threads exit conventionally (return from their procedure). The OS terminates the process that has no threads. All the d'tors are executed.
Some threads either exit via ExitThread or killed by TerminateThread. The automatic objects of those threads are not d'tructed.
Process exited by ExitProcess. Automatic objects are not destructed, global may be destructed (this happens in the CRT is used in a DLL)
Process is terminated by TerminateProcess. All d'tors are not called.
I suggest you check if the DeleteFile (or CFile::Remove that wraos it) is called indeed, and check also if it succeeds. For instance you may open the same file twice for whatever reason
In a couple of scripts that I use I have problem that is intermittent.
Sometimes the script fails when trying to delete a file. According to the error log due to the file being accessed by an other process. I'm guessing that windows not had time to release the file after the previous operation performed on the file ended.
What amount of time would be a good guesstimate after which windows should have had time to release the file again?
If the Windows app is done working with the file it should be closed instantly, because presumably they closed their file handles. There is no delay in time to unlock a file after a file close operation.
If a program forgets to close their file handles though, but ends, Windows will free it for them (just not instantly). Usually it's not long but it can be any amount of time, I haven't seen it take longer than a couple seconds. But proper cleanup should be done to avoid it being locked.
It's also worth mentioning that not all programs open files in a locked way. They can open file specifying what type of access they'd like to give other processes, and they can also lock portions of the file. They may open the file with full read/write permissions to other processes.
If you have no control over the process that is not closing its file handles, but you need to execute it, you could write some kind of loop to keep trying the file for a few seconds.
As another user has posted, it should be done instantly if the file has been closed correctly - with an indetminate delay until the OS sorts it out otherwise...
Always, always dispose of resources correctly.
UNIX file-locking is dead-easy: The operating system assumes that you know what you are doing and lets you do what you want:
For example, if you try to delete a file which another process has opened the operating system will usually let you do it. The original process still keeps it's file-handles until it terminates - at which point the the file-system will quietly re-cycle the disk-resources. No fuss, that's the way I like it.
How different things are on Windows: If I try to delete a file which another process is using I get an Operating-System error. The file is untouchable until the original process releases it's lock on the file. That was great back in the single-user days of MS-DOS when any locking process was likely to be on the same computer that contained the files, however on a network it's a nightmare:
Consider what happens when a process hangs while writing to a shared file on a Windows file-server. Before the file can be deleted we have to locate the computer and ID the process on that computer which originally opened the file. Only then can we kill the process and delete our unwanted file.
What a nuisance!
Is there a way to make this better? What I want is for file-locking on Windows to behave a like file-locking in UNIX. I want the operating system to just let me do what I want because I'm in charge and I know what I'm doing...
...so can it be done?
No. Windows is designed for the "average user", that is people who don't understand anything about a computer. Therefore, the OS tries to be smart to avoid PEBKACs. To quote Bill Gates: "There are no issues with Windows that any number of people want to be fixed." Of course, he knows that 99.9999% of all Windows users can't tell whether the program just did something odd because of them or the guy who wrote it.
Unix was designed when the world was more simple and anyone close enough to a computer to touch it, probably knew how to assemble it from dirty sand. Therefore, the OS usually lets you do what you want because it assumes that you know better (and if you didn't, you will next time).
Technical answer: Unix allocates an "i-nodes" if you create a file. I-nodes can be shared between processes. If two processes create the same file (that is, two processes call create() with the same path), then you end up with two i-nodes. This is by design. It allows for a fancy security feature: You can create files which no one can open but yourself:
Open a file
Delete it (but keep the file handle)
Use the file any way you like
Close the file
After step #2, the only process in the universe who can access the file is the one who created it (unless you want to read the hard disk block by block). The OS will keep the data alive until you either close the file or your process dies (at which time Unix will clean up after you).
This design is the foundation of all Unix filesystems. The Windows file system NTFS works much the same way but the high level API is different. Many applications open files in exclusive mode (which prevents anyone, even backup programs) to read the file. This is even true for applications which just display information like PDF viewers.
That means you'll have to fix all the Windows applications to achieve the desired effect. If you have access to the source, you can create a file in a shared mode. That would allow other processes to access it at the same time but then, you will have to check before every read/write if the file still exists, whether someone has made changes, etc.
According to MSDN you can specify to CreateFile() 3rd parameter (dwSharedMode) shared mode flag FILE_SHARE_DELETE which:
Enables subsequent open operations on a file or device to request delete access.
Otherwise, other processes cannot open the file or device if they request delete access.
If this flag is not specified, but the file or device has been opened for delete access, the function fails.
Note Delete access allows both delete and rename operations.
http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx
So if you're can control your applications you can use this flag.
Note that Process Explorer allow for force closing of file handles (for processes local to the box on which you are running it) via Handle -> Close Handle.
Unlocker purports to do a lot more, and provides a helpful list of other tools.
Also deleting on reboot is an option (though this sounds like not what you want)
That doesn't really help if the hung process still has the handle open. It won't release the resources until that hung process releases the handle. But anyway, in Windows it is possible to force close a file out from under a process that's using it. Process Explorer from sysinternals.com will let you look at and close handles that a process has open.