Most of the vendors that FTP files to us we have the vendor place a file on our FTP server as one extension and after the PUT is complete rename the file to *.FTP. This signals to our process that the file is completely downloaded and our custom process will pick up the file and automatically process it with defined scripts.
I have this new vendor who state that they are not writing the FTP script and cannot tell their system to rename the file after the PUT takes place. So my question, is there a way with a .bat file to check to see if end of file exists?
The reason for doing this is that there's a potential that our custom process may pick up the file and process it before it's fully loaded on the ftp server.
Any thoughts?
Wait until the file's last modified time is at least ten minutes in the past. Alternatively, wait until the file's size hasn't changed for five minutes.
might be a bit of a hacky solution, but could you get the size of the file, then check again after a wait period, if it changes don't process. Update initial value?
Related
I'm looking for a method which would allow me to automatically open a modified file in windows. In other words, something running in the background which detects changes in a given set of files; such that when one is detected, the file is opened. I attempted writing a batch file which saves the last modified date and time to a text file and repeatedly checks that. I think this method works, but I didn't know if there's a better way out there.
My motivation is that I'm scp'ing files regularly from a linux machine to windows, and it would be neat if they just opened automatically on my windows machine after updating locally.
I have 2 applications running in parallel, both doing the following:
check for file not containing "processed"
process the file and then rename it to filename+processed
for every file, only one application shall use it (on a first come first served basis)
I get the files and I also lock them so the other application cannot process it. But when it comes to renaming the file I get a problem. To rename the file, wanted to use the File.renameTo function. However, for that to work, I have to release the lock on the file. But when I release the lock another process may try to use the file. Exactly that should not happen.
Is there any way to prevent the application B from using the file between application A releasing the lock and finishing renaming the file?
EDIT
Some more information:
File creation if the file doesn't exist has to be prevented.
The file will be processed RandomAccessFile (with read and write permission; this creates a new file if it doesn't exist).
Note: On linux, one can rename a file that is locked, so this problem doesn't occur there. However, on Windows a locked file cannot be renamed; I have to release the lock, then rename it. But the time, during which the lock is released creates enables other applications to see that the file is available and then they will try to use it.
Windows applications can do this using the SetFileInformationByHandle function, which allows you to rename the file using the handle you already have open. You probably can't do this natively from Java.
However, a more straightforward solution would be to rename the file (to filename+processing, for example) before you start processing it. Whichever process successfully renames the file in this way is the one responsible for processing it and eventually renaming it to filename+processed.
When I copy a file with CopyFile, will the source file be locked in any way?
Say - if the copy operation starts, and then someone wants to open the source file, will it succeed? What if the "someone" requests exclusive access to the file, will his Win32 API call (probably a CreateFile) fail, or will it wait (or whatever) until the copy operation is done?
If somebody requests exclusive access to a file that's already opened, that request will fail. Windows does not have a file open mode that consists of, "If somebody else is using the file, bump him off."
As I recall, CopyFile opens the source file so that others can read, but not write it.
I'm currently in working on a script to create a custom backup script, the only piece I'm missing is a file monitor. I need some form of a script that will monitor a folder for file changes, and then run a command with the file that's changed.
So, for example, if the file changes, it'll execute "c:/syncbatch.bat %Location_Of_File%"
In VBScript, you can monitor a folder for file changes by subscribing to the WMI __InstanceModificationEvent event. These articles contain sample code that you can learn from and adapt to your specific needs:
WMI and File System Monitoring
How Can I Monitor for Different Types of Events With Just One Script?
Calling WMI is fairly cryptic and it causes the WMI service to start running which can contribute to bloat since its fairly large and you really can't cancel the file change notifications you've requested from it without rebooting. Some people experimenting with remote printing from a Dropbox folder found that a simple VBScript program that ran an endless loop with a 10 second WScript.Sleep call in the loop used far less resource. Of course to stop it you have to task kill that script or program into it some exit trigger it can find like a specifically named empty file in the watch folder, but that is still easier to do than messing with WMI.
The Folder Spy http://venussoftcorporation.blogspot.com/2010/05/thefolderspy.html
is a free lightweight DOT.NET based file/folder watching GUI application I'ved used before to run scripts based on file changes. It looks like the new version can pass the event filename to the launched command. The old version I had didn't yet support file event info so when launched, my script had to instance a File System Object and scan the watched folder to locate the new files based on criteria like datestamps and sizes.
This newer version appears to let you pass the file name to the script if you say myscript.vbs "*f" on the optional script call entry. Quotes can be important when passing file paths that have spaces in folder names. Just remember if you are watching change events you will get a lot of them as a file grows or is edited, usually you just want notification of file adds or deletes.
Another trick your script can do is put the file size in a variable, sleep for a few seconds, and check the file again to see if its changed. if it hasn't changed in a few seconds you can usually assume whatever created it is done writing it to disk. if it keeps changing just loop until its stable.
Our issue is that our project has files being downloaded using wget to the file system. We are using ruby to read the downloaded files for data.
How is it possible to tell if the file is completely downloaded so we don't read a half complete file?
I asked a very similar question and got some good answers... in summary, use some combination of one or more of the following:
download the file to a holding area and finally copy to your input directory;
use a marker file, created once the download completes, to signal readiness;
poll the file twice and see if its size has stopped increasing;
check the file's permissions (some download processes block reads during download);
use another method like in-thread download by the Ruby process.
To quote Martin Cowie, "This is a middleware problem as old as the hills"...
The typical approach to this is to download the file to a temporary location and when finished 'move' it to the final destination for processing.