I need to back up some large files that are being written to disk by a process. The process is perpetually running, and occasionally dumps large files that need to be moved over the network. Having the process do this itself is not an option, as the process locks out users whilst it is doing file dumps.
So, this runs under a windows machine, and as a primarily linux user, I am not entirely certain how to do this...
Under linux I would simply use a cron job in the folder (I know the glob that will match the output files), then check lsof, to ensure that the file is not being written to, such that I don't try to copy a partially complete file. Data integrity is critical, so I would normally md5 the files before and after the copy.
So I guess my question is -- how does one do this sort of stuff under windows? I feel like I am kneecapped from the start -- I can use python, but I can't emulate lsof, nor cron to do the task scheduling.
I tried looking at "handle" -- but it needs admin privelidges at execution time, which is also not an option. I can't run the backup process as an admin, it has to run with user privs.
Thanks..
Edit: I just realised I could keep the python instance running, with a sleep, so task scheduling is not a problem :)
For replacing cron you can use the "Task Scheduler" in windows to start your script every few minutes (or specific times).
For lsof the question was discussed here : How can I determine whether a specific file is open in Windows?
Related
I am writing a Go app targeted at macOS and Windows and that needs to monitor what processes write to a file at a given path. More specifically, I need to verify that only one specific process writes to the file for the duration that my program is running. On macOS, I can monitor the file via the built-in fs_usage command. Does anyone have an idea for how to achieve equivalent monitoring on at least Windows 10 and later without requiring the user to install any additional software.
Note that I don't expect for there to exist a pure Go solution and I don't mind interoperating to achieve the desired result.
I'm developing an app. The operating system I'm using is linux. I need to run if possible a ruby script on the file created in the directory. I need to keep this script always running. The first thing I thought about is inotify:
The inotify API provides a mechanism for monitoring file system events. Inotify can be used to monitor individual files, or to monitor directories.
It's exactly what I need, then I found "rb-inotify", a wrapper fir inotify.
Do you think there is a better way of doing what I need than using inotify? Also, I really don't understand the way that I have to use rb-inotify.
I just create, for example, a rb file with:
notifier = INotify::Notifier.new
notifier.watch("directory/to/check",:create) do |event|
#do task with event.name file
end
notifier.run
Then I just ruby myRBNotifier.rb, and it will stay looping for ever. How do I stop it? Any idea? Is this a good approach?
I'd recommend looking at god. It's designed for this sort of task, and makes it pretty easy to build a monitoring system for background and daemon apps.
As for the main code itself, inotify isn't cross-platform, so if you have a possibility you'll need to run on Windows or Mac OS then you'll need a different solution. It's not too hard to write a little piece of code that checks your target directory periodically for a change. If you need to know what changed, read and cache the directory entries then compare them the next time your code runs. Use sleep between runs to wait some period of time before looping.
The old-school method of doing similar things is to use cron to fire off a job at regular intervals. That job can be your script that checks whether the file list changed by comparing it to the cached version, then acting as needed if something is different.
Just run your script in the background with
ruby myRBNotifier.rb &
When you need to stop it, find the process id and use kill on it:
ps ux
kill [whatever pid your process gets from the OS]
Does that answer your question?
If you're running on a mac/unix machine, look at the launchctl man page. You can set up a process to run and execute a ruby script whenever a file changes. It's highly configurable.
How can I start my program automatically if it crashes on windows 2003 server? Sometimes my program just crashes, is there a way in windows or settings that I can set?
There are several ways to create a process supervisor/guardian process on Windows.
First, is to leverage windows command line capabilities. Create a bat file:
#echo off
:start
start /w "your app to watch.exe"
goto start
start /w will wait for the process to exit. When the process crashes and exits, the bat script will relaunch it.
Another option is to use free supervisor tool https://github.com/chebum/Supervisor. It allows to restart the crashed app, plus it allows to monitor two or more apps at once and it will automatically close these apps when supervisor's window is closed.
The usual approach is to run what is known as a guardian process. This is a separate process, often a service, that monitors the state of the main process. When the guardian detects that the main service has died, it re-spawns it.
To the very best of my knowledge, there is not built in Windows functionality to do this for you.
Notice: running self-looping bat files can be useful, but unless you know what you're doing, they can wreak all kinds of havoc. This goes especially if you run them on startup. You have been warned.
Anyway. I just remembered something from my 286 days, when I played around a lot with BAT files. If you write the file
yourprogram.exe
some other event
the BAT file will run yourprogram, and then pause and wait around in the background until the program exits. After that it will run "some other event". This used to be kind of annoying if you wanted to run multiple things at once, but here it's actually useful. Using this, it's possible to make it run a loop that restarts the program (and reruns the bat file) as soon as it exits. Combine this with https://superuser.com/questions/62525/run-a-completly-hidden-batch-file, and you'll never even see it happening.
The final BAT file ("restart.bat" in this example) will look something like:
c:\[location]\yourprogram.exe
wscript "C:\[location]\invisible.vbs" "C:\[location]\restart.bat"
That's about it. Start the program (on startup via task or even just startup folder) with line 2, and this ought to solve your problem :)
Oh, if you want to stop the loop, just rename the bat file or put "// " in front of the two lines, save it, and exit the program.
If the program you are running requires admin rights, the solution I found was using psexec (http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx) to run both the program and the bat with elevated privileges. In that case the BAT will look like:
c:\[location]\psexec -h c:\[location]\yourprogram.exe
c:\[location]\psexec -h wscript "C:\[location]\invisible.vbs" "C:\[location]\restart.bat"
Then you run the bat as administrator, or run the second line (without the psexec part) from task scheduler with elevated privileges. BEWARE: running it as a normal user and clicking "no" on the UAC prompt gave me a BSOD, probably because it looped "can't run program because of lacking privileges" a couple of billion times or something :)
You can use RegisterApplicationRestart.
"If you register for restart and the application encounters an
unhandled exception or is not responsive, the user is offered the
opportunity to restart the application; the application is not
automatically restarted without the user's consent. "
For automatic restart without user intervention, there is also RestartOnCrash. Works with all Windows versions.
I was looking for something similar. There are two options to handle this - either you can write a small script by yourself or use something that is already existing.
After some googling I came across this nice list. The blogger has compiled about 8 tools to automatically restart a crashed or closed application.
Unfortunately there are no settings in Windows to automatically restart a regular program when it crashes.
Do you need to actively interact with your application's GUI? Some of the Service Wrappers (designed to run any application as a Windows Service) will monitor your application and restart it when it fails, but be sure investigate Session 0 Isolation to ensure that it won't get in the way.
You may use some special app like BDV SystemEvents or any other. It allows you to specify application which will be started if some another application is closed. Specify the same application as a Condition and as an Action and you will get expected results.
I decided to write a program in RUBY in which the following things should be done:
1 - this program must run a specific program (for example utorrent) at a specific time (for example 1 pm).
2 - this program must turn off my computer at another specific time.
I don't have any idea about the algorithm and manner of writing such program.
One of the easiest ways to do this is to simply send kill signals to the processes, requesting the app shut down normally (Linux), or in Windows use taskkill.
To shutdown a machine in Windows, you can use shutdown /s /f which forcibly closes any programs that are running, and turns the computer off.
No matter which way you do it, you'll basically be running the enter link description heresystem() command in Ruby, which runs command line commands. To make your app portable, you simply look up how to do these tasks in each target OS, and you're done.
Two more alternatives that work the same as your Ruby proposal, but which are not as easily portable:
Write a batch file in Windows that calls taskkill, or a bash script on Linux. Unless the program in question provides a specific way to shut it down via its own command-line parameters, this should work for any/all applications.
You can also use Task Scheduler in Windows, or cron in Linux to do the same thing.
UNIX file-locking is dead-easy: The operating system assumes that you know what you are doing and lets you do what you want:
For example, if you try to delete a file which another process has opened the operating system will usually let you do it. The original process still keeps it's file-handles until it terminates - at which point the the file-system will quietly re-cycle the disk-resources. No fuss, that's the way I like it.
How different things are on Windows: If I try to delete a file which another process is using I get an Operating-System error. The file is untouchable until the original process releases it's lock on the file. That was great back in the single-user days of MS-DOS when any locking process was likely to be on the same computer that contained the files, however on a network it's a nightmare:
Consider what happens when a process hangs while writing to a shared file on a Windows file-server. Before the file can be deleted we have to locate the computer and ID the process on that computer which originally opened the file. Only then can we kill the process and delete our unwanted file.
What a nuisance!
Is there a way to make this better? What I want is for file-locking on Windows to behave a like file-locking in UNIX. I want the operating system to just let me do what I want because I'm in charge and I know what I'm doing...
...so can it be done?
No. Windows is designed for the "average user", that is people who don't understand anything about a computer. Therefore, the OS tries to be smart to avoid PEBKACs. To quote Bill Gates: "There are no issues with Windows that any number of people want to be fixed." Of course, he knows that 99.9999% of all Windows users can't tell whether the program just did something odd because of them or the guy who wrote it.
Unix was designed when the world was more simple and anyone close enough to a computer to touch it, probably knew how to assemble it from dirty sand. Therefore, the OS usually lets you do what you want because it assumes that you know better (and if you didn't, you will next time).
Technical answer: Unix allocates an "i-nodes" if you create a file. I-nodes can be shared between processes. If two processes create the same file (that is, two processes call create() with the same path), then you end up with two i-nodes. This is by design. It allows for a fancy security feature: You can create files which no one can open but yourself:
Open a file
Delete it (but keep the file handle)
Use the file any way you like
Close the file
After step #2, the only process in the universe who can access the file is the one who created it (unless you want to read the hard disk block by block). The OS will keep the data alive until you either close the file or your process dies (at which time Unix will clean up after you).
This design is the foundation of all Unix filesystems. The Windows file system NTFS works much the same way but the high level API is different. Many applications open files in exclusive mode (which prevents anyone, even backup programs) to read the file. This is even true for applications which just display information like PDF viewers.
That means you'll have to fix all the Windows applications to achieve the desired effect. If you have access to the source, you can create a file in a shared mode. That would allow other processes to access it at the same time but then, you will have to check before every read/write if the file still exists, whether someone has made changes, etc.
According to MSDN you can specify to CreateFile() 3rd parameter (dwSharedMode) shared mode flag FILE_SHARE_DELETE which:
Enables subsequent open operations on a file or device to request delete access.
Otherwise, other processes cannot open the file or device if they request delete access.
If this flag is not specified, but the file or device has been opened for delete access, the function fails.
Note Delete access allows both delete and rename operations.
http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx
So if you're can control your applications you can use this flag.
Note that Process Explorer allow for force closing of file handles (for processes local to the box on which you are running it) via Handle -> Close Handle.
Unlocker purports to do a lot more, and provides a helpful list of other tools.
Also deleting on reboot is an option (though this sounds like not what you want)
That doesn't really help if the hung process still has the handle open. It won't release the resources until that hung process releases the handle. But anyway, in Windows it is possible to force close a file out from under a process that's using it. Process Explorer from sysinternals.com will let you look at and close handles that a process has open.