I need to monitor a directory for new files that are added (constantly: one or two every 5 seconds are added and deleted).
It would be nice to tail them constantly in terminal, but since this seems unlikely, it would be also good to append all of the tails to a file. That way when I stop my process I can review all of the files that were added and deleted (I need the contents not the filename).
A shell script will work as well as long as it will run on Mac OSX Lion.
Any help? I've tried multi-tail and tail -f (but these don't monitor the directory for new files).
Maybe xtail ?
It could be installed with homebrew or macports on osx.
CoreServices provides methods for watching changes to directories.
Apple's Carbon documentation is excellent, except that I can't seem to link to a specific function. Search for FNSubscribeByPath.
If you really need to look at the contents of the files that get added, then you can write a tool that uses FNSubscribeByPath to tell you when the directory changes and what changed, then tail -f the file.
You should probably have something that keeps track of what "tail -f" instances have been launched, so that it can kill them when the files are deleted. Otherwise, your process table will eventually fill up.
You should be able to use logstash for this. It will give you a stream of events for a set of inputs (e.g. files) and give you a stream of data you can push to another output (e.g. a file). It will also give you a lot of options for future growth, while being easy to setup in the short term.
http://www.logstash.net/
Related
I am trying to keep two directories synchronized with the same files in them.
Files are dropped into Directory A throughout the day. I would like to create a file watcher script that will copy files from Directory A to Directory B as soon as they are dropped.
My thought was to run the job every minute and simply copy everything that dropped in the last minute, but I am wondering if there is a better solution out there.
I'm running MKS toolkit under Windows. Different servers, same operating system.
Thanks for your help!
If you use Linux, you can hook into the kernel using the inotify API to get notified if something in a folder changes. There are command line versions like inotifywatch(1) as well.
To copy the files, I suggest to use rsync(1): it is clever, knows how to clean up after itself and it will create new files hidden while they are copied so users and programs are less likely to pick them up before they are complete.
There are a lot of similar posts out there, but none that does exactly what I want: I want a simple script, that watches a folder on a Mac for changes and passes the filename of the modified file to command/script when changes are detected.
The tool inotifywait doesn't exist on a mac. Folder events or fswatch or watchdog seem to be getting close, but it turns out they only watch folders or only return the folder name. watchdog might be doing what I need, but I could only find commands like this
watchmedo shell-command --recursive --command='echo ${watch_src_path}' .
which again only passes the folder name in the variable watch_src_path. Does anyone know how to get it (or another program) to pass on the file name of the changed file?
Even if the question is very old, it may be useful to other people looking for the same functionality. I wrote fsw exactly to fill this gap. fsw will give you the name of the changed paths and the type of change event.
Hope this helps.
Edited: fsw was merged with fswatch.
I want to monitor a directory (of thousands of files, with about 5 levels of sub directories) for when files are changed. I know I can use the FSEvents API to monitor a directory for when files change inside that directory, but I can't seem to figure out how to determine which file(s) changed. This reference suggests I build a binary tree and traverse the tree each time an event is triggered, is that the best way to determine which files were changed? If not, what are some better alternatives?
Is it better to recursively scan the directory and attach kqueue to every file? I'm not sure how well that would work on thousands of files?
I've used UKKQueue before with mixed results.
I've recently become aware of a better solution, but I haven't tried this. If you only need to target Lion, I think that the new best-practice way to do this is to use an NSFileCoordinator. You implement the methods of NSFilePresenter to indicate which directory you're interested in (the presentedItemURL property) and the system will notify you when a sub item moves/changes/is deleted (via methods like presentedSubitemDidChangeAtURL:)
I'd love to hear how that works out if you do go that route.
If you create your stream using kFSEventStreamCreateFlagFileEvents then you will get events for the changes to each file rather than just a notification of the change to the watched directory. Unfortunately this is only available in OSX 10.7 and later.
I'm trying to reverse-engineer a program that does some basic parsing: text in, text out. I've got an executable "reference implementation" and the source code to what must be a different version, since the compiled source output != executable output.
The process creates and deletes temporary files very quickly in a multi-step parsing process. If I could take a look at the individual temporary files, I could get some great diagnostic data to narrow down where my source differs from the binary.
Is there any way to do any of the following?
Freeze a directory so that file creation will work but file deletion will fail silently?
Run a program in "slow motion" so that I can look at the files that it creates?
Log everything that a program does, including any data written out to files?
Running a tool like NTFS Undelete should give you the chance to recover the temporary files it's creating then deleting. Combine this with ProcMon from Sysinternals to get the right filenames.
You didn't mention what OS you're doing this on, but assuming you're using Windows...
You might be able to make use of SysInternals tools like Process Explorer and Process Monitor to get a better idea of the files being accessed. As far as I know, there's no "write-only" option on folders. For "slowing down" the files, you'd just need to use a slower computer. For logging, the SysInternals tools will help out quite a bit. Once you have a file name(s) that are being created, you could try preventing their deletion by opening the files in a stream from another process. That would prevent the system from being able to delete them.
There are two ways to attack this:
Run various small test cases through both systems and notice the differences. Since the test cases are small, you should be able to figure out why your code works differently than the executable.
Disassemble the executable and remove all the "delete temp file" instructions. Depending on how this works, this could be a very complex task (say when there is no central place where it happens).
my app actually goes to different folders and takes each file into account and reads each file and does a lot of processing on them and marks the folder it has processed as done. but this is not happening as the system is immediately generating files like .DS_store and .localized and .trash. so is there any mechanism to skip processing hidden files or stop the os from generating these files programatically?
Thanks
Couldn't you change your app to just ignore files that start with "."? You've tagged this Cocoa, so using something like NSFileManager's contentsOfDirectoryAtURL:includingPropertiesForKeys:options:error: seems appropriate. One of the options you can specify is NSDirectoryEnumerationSkipsHiddenFiles, which will skip hidden files.
Check the documentation for more details.
I'm not aware of any option that disables the generation of .DS_Store files locally. There is an option for remote, here.
Another way to do it could be to create a unix user for just that job and let him own the dirs, so that the Finder never can go there. Either start the job manually using sudo, or make it a setuid job.. or use launchd.