Deleting a locked file created with TempFile - windows

I have a GUI (lxn/walk) app patcher that downloads a file via ftp, streams it to a temporary file and extracts the contents to update the local files. The remove file command is deferred.
This works unless the user exits the program while the file is downloading, then the file isn’t deleted.
I tried to fix this by doing a graceful exit by catching the signal and removing the file there. But unfortunately it throws an error that the file can’t be deleted because it is being used by another program. Which makes sense because the another program is actually itself still writing to the temporary file.
Now I’m stuck and don’t know what to do to make sure that the temporary file is automatically gone once the patcher is not running. How do I do that correctly?
The file could also be created as a normal file, not just a temp file. I would just like to ask too, where in windows is best to write a temporary file?


Now I’m stuck and don’t know what to do to make sure that the temporary file is automatically gone once the patcher is not running. How do I do that correctly?
There are no guaranteed ways to accomplish this as many things beyond the control of the application can cause it to exit. A power failure or kernel panic due to some hardware issue can crash the machine or force it to be restarted.
A strategy that is in common use is to implement a check on program startup for the status of the previous run. Some applications create a lock file at start and remove it on graceful exit. If this lock file exists when the program is restarted, this means the previous run did not result in a clean exit, and the application can take any corrective action. The exact action to be taken depends on the nature of the application, some refuse to start, others give warnings to users.
I would just like to ask too, where in windows is best to write a temporary file?
Each OS has its own location for temporary files. If you eliminate the dir argument to TempFile, it will create it in the appropriate location, as mentioned in the documentation:
TempFile creates a new temporary file in the directory dir, opens the
file for reading and writing, and returns the resulting *os.File. The
filename is generated by taking pattern and adding a random string to
the end. If pattern includes a "*", the random string replaces the
last "*". If dir is the empty string, TempFile uses the default
directory for temporary files (see os.TempDir). Multiple programs
calling TempFile simultaneously will not choose the same file. The
caller can use f.Name() to find the pathname of the file. It is the
caller's responsibility to remove the file when no longer needed.
From os.TempDir we see the following:
On Unix systems, it returns $TMPDIR if non-empty, else /tmp. On
Windows, it uses GetTempPath, returning the first non-empty value
from %TMP%, %TEMP%, %USERPROFILE%, or the Windows directory. On
Plan 9, it returns /tmp.
The directory is neither guaranteed to exist nor have accessible
permissions.

Related

How to know if FSEvents event corresponds to new file still being written to

In my backup app I am monitoring changes to a user-selected directory A and mirroring all file changes to another directory B. The only problem seems to be: how do I know if a file has been created and is still being written to, such as when an app creates a very big file or the Finder copies a file?
I would like to ignore this event because another event will follow corresponding to the moment in which the write operation has finished and it is almost certainly safe to copy the file from A to B (starting a copy while the original file is still being written usually results in an error). I tried to analyze the flags passed to the FSEvents callback, but it seems like different applications cause different flags to be passed (for instance the cp and touch commands, a copy initiated in the Finder, creating a file from TextEdit and Word all result in different flags).

Determine if file can be deleted on windows in go

Is there any reasonable way to determine if a file can be successfully deleted on the Windows platform using the go language?
I've tried using various flags calling os.OpenFile including using os.O_TRUNC all the calls are successful yet remove will fail when the file is in use.
I've also tried using the locking mechanisms specific to Windows (i.e. kern.NewProc("LockFileEx") etc) and these seem to get exclusive locks successfully yet os.Remove (after unlocking) will fail.
I realize that if between the test and the remove call if another process opens the file delete will still fail but this isn't the case here, with my tests I know the file is in use yet I can't find a test to tell me that (other than actually calling os.Remove)

On Windows, the Go os.Remove function uses the Windows DeleteFile function. You must satisfy certain conditions for a Windows file to be deleted. For example, "The DeleteFile function marks a file for deletion on close. Therefore, the file deletion does not occur until the last handle to the file is closed", "The DeleteFile function fails if an application attempts to delete a file that has other handles open for normal I/O", and so on.

Script to watch for file changes & if detected, backup the original first.

As per the title I'm looking for a way to backup files (SQL queries to be specific) when they are saved over/replaced.
This is to prevent accidental saving without a decent fall back.
I think that using a combination of Powershell and .NET FileSystemWatcher might be a good combo and instead of an alert I could take the original and append datetime to it in another location.
(https://gallery.technet.microsoft.com/scriptcenter/Powershell-FileSystemWatche-dfd7084b)
So my question is, how to interrupt that save process within Windows and copy the file out before it's saved over?
I'm thinking it might not be possible without calling the PS when a save is initiated on that specific directory.
I have access to PS, VBS and elevated cmd.
TIA for any pointers or further reading.

There's no way to backup/save the previous before a newer version overwrites it as there are no OnBeforeX events in the FileSystemWatcher.
Changed: Occurs when a file or directory in the specified Path is changed.
Created: Occurs when a file or directory in the specified Path is created.
Deleted: Occurs when a file or directory in the specified Path is deleted.
Disposed: Occurs when the component is disposed by a call to the Dispose method. (Inherited from Component.)
Error: Occurs when the instance of FileSystemWatcher is unable to continue monitoring changes or when the internal buffer overflows.
Renamed: Occurs when a file or directory in the specified Path is renamed.
The only way to do this would be to perform an initial back up of all the files. Then backup, with a timestamp pre/suffix, after each Changed event.

Rename a file that multiple processes are trying to use

I have 2 applications running in parallel, both doing the following:
check for file not containing "processed"
process the file and then rename it to filename+processed
for every file, only one application shall use it (on a first come first served basis)
I get the files and I also lock them so the other application cannot process it. But when it comes to renaming the file I get a problem. To rename the file, wanted to use the File.renameTo function. However, for that to work, I have to release the lock on the file. But when I release the lock another process may try to use the file. Exactly that should not happen.
Is there any way to prevent the application B from using the file between application A releasing the lock and finishing renaming the file?
EDIT
Some more information:
File creation if the file doesn't exist has to be prevented.
The file will be processed RandomAccessFile (with read and write permission; this creates a new file if it doesn't exist).
Note: On linux, one can rename a file that is locked, so this problem doesn't occur there. However, on Windows a locked file cannot be renamed; I have to release the lock, then rename it. But the time, during which the lock is released creates enables other applications to see that the file is available and then they will try to use it.

Windows applications can do this using the SetFileInformationByHandle function, which allows you to rename the file using the handle you already have open. You probably can't do this natively from Java.
However, a more straightforward solution would be to rename the file (to filename+processing, for example) before you start processing it. Whichever process successfully renames the file in this way is the one responsible for processing it and eventually renaming it to filename+processed.

Is it safe to recompile an executable while it's running?

What happens if I recompile an executable while it's running? Does the operating system read all of the executable's contents into memory when it starts running it, so it will never read the new executable file? Or will it read sections of the new executable file thinking it hasn't changed, leading to possibly undefined behaviour?
What if I have a script running which repeatedly invokes an executable in a loop, and I recompile the executable while the script is running. Is it guaranteed that future iterations of the loop will invoke the new executable, and only the result of the invocation that was in progress when the switch was made might be corrupted?
My OS is Linux, but I'm also curious about what happens on Windows.

Since this is a conventional compiler, that writes out an executable file, let's follow it in Linux.
The first thing to know is that a Linux filename doesn't directly refer to the file, but rather to a directory entry, which is independent of the filename. A file doesn't actually need to have a filename, but if it doesn't it will be difficult to refer to it.
If a process is using a file, and you replace or delete it, the process will continue using that file through its directory entry. Any new process using the file, or looking it up, will get the new version (if you replaced it) or fail to find it (if you deleted it). Once all the processes are through with the old file, it will be deleted from the file system.
Therefore, if you recompile and create a new executable of the same name, you won't affect the running process. It will continue to use the old executable. Any new process that tries to open the file will get the new one. If you've got system("foo"); in a loop, each time it executes it it will see what the filename foo means right then.
Windows handles files differently. In general, if there's a process using a file, the file is locked and may not be deleted or replaced.

It depends.
If the OS read the whole of the executable into memory and doesn't refer back to the disk image then yes you can recompile it while it was "in use".
In practice this doesn't always happen. If the OS keeps a file handle open (like Windows does) on the executable this will prevent the file being deleted and/or overwritten.
With Linux/Unix it is possible to overwrite a file that's "in use". See David Thornley's answer for a detailed explanation.

In Windows you can't delete a locked file but what most people don't know is that you can move or rename a running exe.
So you could
move the old exe to a temp directory on the same drive
schedule it for deletion on the next reboot: MoveFileEx(name, NULL, MOVEFILE_DELAY_UNTIL_REBOOT);
move a new exe in its place.
The old program will keep running but new processes will use the new file.

Under Linux, executables are demand paged into memory as needed. The executable on disk becomes the backing store for the application. This means you cannot modify the executable on disk or you will affect a running application. If you try to open(2) an in-use executable for writing, you will get an ETXTBSY (Text file busy) error (check the man page for open(2)).
As many others have said, you can remove the file from the filesystem (unlink(2)) and the kernel will maintain a reference to it and not delete it from disk until there are no more references (when the process exits, it will release its reference to the file). This means you can effectively "overwrite" an in-use executable by first removing it and then creating a new file with the same name as the old file.
So, it comes down to how the compiler creates the executable when "overwriting" an existing file. If it just opens the file for writing and truncates it (O_WRONLY|O_CREAT|O_TRUNC), the it will fail with an ETXTBSY error. If it first removes the existing output file and creates a new one, it will work without error.

I would imagine it wouldn't let you replace the file, since windows locked it down while it was in use.

It depends. From what I've experienced, on Linux you can still be running a program if you delete it (and it's not too large). But I don't think that's defined behavior.
As far as the loop goes, depending on how you're invoking the executable, you will likely end out crashing your script when it goes to execute a program that's only halfway been written.

In Windows you can't if the executable is still running, the file will be locked. If the exe isn't actually running, the new runs should pick up the new one, depending on how your script is coded among other things.
I don't know about Linux.

The Executable might be loaded completely into the memory on startup, however if it's large enough, and running long enough, the OS might decide to swap out some unused parts of it.
Since the OS assumes that the program's file is still there, there is no reason to actually write these memory blocks into the swap file. So they are simply invalidated and reused. If the program needs these pages again, the OS loads them from the executable file.
In Windows this actually happens automagically, since a loaded module is a memory mapped file. That also means that the file is locked during it's execution, and you will not be able to overwrite it easily.
Not sure about Linux, but IIRC it does the swapping the same way.

Resources