Bash script to check if a new file has been created on a directory after run a command - bash

By using bash script, I'm trying to detect whether a file has been created on a directory or not while running commands. Let me illustrate the problem;
#!/bin/bash
# give base directory to watch file changes
WATCH_DIR=./tmp
# get list of files on that directory
FILES_BEFORE= ls $WATCH_DIR
# actually a command is running here but lets assume I've created a new file there.
echo >$WATCH_DIR/filename
# and I'm getting new list of files.
FILES_AFTER= ls $WATCH_DIR
# detect changes and if any changes has been occurred exit the program.
After that I've just tried to compare these FILES_BEFORE and FILES_AFTER however couldn't accomplish that. I've tried;
comm -23 <($FILES_AFTER |sort) <($FILES_BEFORE|sort)
diff $FILES_AFTER $FILES_BEFORE > /dev/null 2>&1
cat $FILES_AFTER $FILES_BEFORE | sort | uniq -u
None of them gave me a result to understand there is a change or not. What I need is detecting the change and exiting the program if any. I am not really good at this bash script, searched a lot on the internet however couldn't find what I need. Any help will be appreciated. Thanks.

Thanks to informative comments, I've just realized that I've missed the basics of bash script but finally made that work. I'll leave my solution here as an answer for those who struggle like me.:
WATCH_DIR=./tmp
FILES_BEFORE=$(ls $WATCH_DIR)
echo >$WATCH_DIR/filename
FILES_AFTER=$(ls $WATCH_DIR)
if diff <(echo "$FILES_AFTER") <(echo "$FILES_BEFORE")
then
echo "No changes"
else
echo "Changes"
fi
It outputs "Changes" on the first run and "No Changes" for the other unless you delete the newly added documents.

I'm trying to interpret your script (which contains some errors) into an understanding of your requirements.
I think the simplest way is simply to rediect the ls command outputto named files then diff those files:
#!/bin/bash
# give base directory to watch file changes
WATCH_DIR=./tmp
# get list of files on that directory
ls $WATCH_DIR > /tmp/watch_dir.before
# actually a command is running here but lets assume I've created a new file there.
echo >$WATCH_DIR/filename
# and I'm getting new list of files.
ls $WATCH_DIR > /tmp/watch_dir.after
# detect changes and if any changes has been occurred exit the program.
diff -c /tmp/watch_dir.after /tmp/watch_dir.before
If the any files are modified by the 'commands', i.e. the files exists in the 'before' list, but might change, the above will not show that as a difference.
In this case you might be better off using a 'marker' file created to mark the instance the monitoring started, then use the find command to list any newer/modified files since the market file. Something like this:
#!/bin/bash
# give base directory to watch file changes
WATCH_DIR=./tmp
# get list of files on that directory
ls $WATCH_DIR > /tmp/watch_dir.before
# actually a command is running here but lets assume I've created a new file there.
echo >$WATCH_DIR/filename
# and I'm getting new list of files.
find $WATCH_DIR -type f -newer /tmp/watch_dir.before -exec ls -l {} \;
What this won't do is show any files that were deleted, so perhaps a hybrid list could be used.

Here is how I got it to work. It's also setup up so that you can have multiple watched directories with the same script with cron.
for example, if you wanted one to run every minute.
* * * * * /usr/local/bin/watchdir.sh /makepdf
and one every hour.
0 * * * * /user/local/bin/watchdir.sh /incoming
#!/bin/bash
WATCHDIR="$1"
NEWFILESNAME=.newfiles$(basename "$WATCHDIR")
if [ ! -f "$WATCHDIR"/.oldfiles ]
then
ls -A "$WATCHDIR" > "$WATCHDIR"/.oldfiles
fi
ls -A "$WATCHDIR" > $NEWFILESNAME
DIRDIFF=$(diff "$WATCHDIR"/.oldfiles $NEWFILESNAME | cut -f 2 -d "")
for file in $DIRDIFF
do
if [ -e "$WATCHDIR"/$file ];then
#do what you want to the file(s) here
echo $file
fi
done
rm $NEWFILESNAME

Related

Create script taking output from another

Basically I want to have a (generated) playlist in every folder containing music, sorted in order and has the same name as the folder.
I have a script that takes the information from music files and creates a playlist in a specific format.
1) I can run the script on single folder by
$ ./playlist.sh ~/Path/To/Music > output.xwaxpls
2) Put the playlist in a way that increases numerically (all my music is formatted %track% - %artist% - %title%)
$ sort -o output.xwaxpls output.xwaxpls
3) Change the name of output.xwaxpls to parent directory name
Unable to get this far yet.
Obviously I would love a script that would do all this.
I'm trying to learn and do each step one at a time but sadly I'm falling at the first hurdle. I can't even work out how to run the playlist.sh on all subdirectories. It's putting me off as I'm stuck before I can try the other two.
I would love some help....
Edit:
Another few hours and I'm getting somewhere !!
`#!/bin/sh
curdir=$(pwd)
for folder in ~/Music/*; do
[ -d "$folder" ] && cd "$folder" && ~/Desktop/./playlist.sh "$folder" > "$folder".xwaxpls && sort -o "$folder".xwaxpls "$folder".xwaxpls
done
`
This creates the playlists that I want, in the format that I want.......but the playlists are saved in ~/Music, not in the folder that contain the files.
I have tried to change the ~/Desktop/./playlist.sh "$folder" > "$folder".xwaxpls part to ~/Desktop/./playlist.sh "$folder" > "$folder"/"$folder".xwaxpls but an error is kicked out saying cannot create /home/user/Music/Singles//home/user/Music/Singles.xwaxpls: Directory nonexistent.
I have also tried to add && mv $playlist.xwaxpls $playlist/$playlist.xwaxpls which results in the same // error.
The // I understand, as I am forcing a double path but why is the xwaxpls file not written to the $folder in the first place ? I presume it's because I have specified the ~/Music/* in the for line ?
You might want something like
#!/bin/bash
for folder in ~/Music/*; do
[ -d "$folder" ] || continue
cd "$folder" || continue
~/Desktop/playlist.sh "$folder" | sort -n > "$folder"/"${folder##*/}".xwaxpls
done
Remember that you do not try sort -o file file, most unix commands don't like to write to an output file that is input at the same time.

gnu parallel to parallelize a for loop

I have seen several questions about this topic, but I lack the ability to translate this to my specific problem. I have a for loop that loops through sub directories and then executes a .sh script on a compressed text file inside each directory. I want to parallelize this process, but I'm struggling to apply gnu parallel.
Here is my loop:
for d in ./*/ ; do (cd "$d" && script.sh); done
I understand I need to input a list into parallel, so i have been trying this:
ls -d */ | parallel cd && script.sh
While this appears to get started, I get an error when gzip tries to unzip one of the txt files inside the directory, saying the file does not exist:
gzip: *.txt.gz: No such file or directory
However, when I run the original for loop, I have no issues aside from it taking a century to finish. Also, I only get the gzip error once when using parallel, which is so weird considering I have over 1000 sub-directories.
My questions are:
How do I get Parallel to work in my case? How do I get parallel to parallelize the application of a .sh script to 1000s of files in their own sub-directories? ie- what is the solution to my problem? I gotta make progress.
What am I missing? Syntax, loop, bad script? I want to learn.
Is Parallel actually attempting to run all these .sh scripts in parallel? Why dont I get an error for every .txt.gz file?
Is parallel the best option for the application? Is there another option that is better suited to my needs?
Two problems:
In:
ls -d */ | parallel cd && script.sh
what is paralleled is just cd, not script.sh. script.sh is only executed once, after all parallel cd jobs have run, if there was no error. It is the same as:
ls -d */ | parallel cd
if [ $? -eq 0 ]; then script.sh; fi
You do not pass the target directory to cd. So, what is executed by parallel is just cd, which just changes the current directory to your home directory. The final script.sh is executed in the current directory (from where you invoked the command) where there are probably no *.txt.gz files, thus the error.
You can check yourself the effect of the first problem with:
$ mkdir /tmp/foobar && cd /tmp/foobar && mkdir a b c
$ ls -d */ | parallel cd && pwd
/tmp/foobar
The output of pwd is printed only once, even if you have more than one input directory. You can fix it by quoting the command and then check the second problem with:
$ ls -d */ | parallel 'cd && pwd'
/homes/myself
/homes/myself
/homes/myself
You should see as many pwd outputs as there are input directories but it is always the same output: your home directory. You can fix the second problem by using the {} replacement string that is substituted with the current input. Check it with:
$ ls -d */ | parallel 'cd {} && pwd'
/tmp/foobar/a
/tmp/foobar/b
/tmp/foobar/c
Now, you should have all input directories properly listed in the output.
For your specific problem this should work:
ls -d */ | parallel 'cd {} && script.sh'

can Linux Bash search for file every 60 seconds and execute file commands? how would I do this?

Basically I want to do something like this from bash.
if a file exists in a directory rename,move,whatever
if it doesn't exist loop every 60 seconds:
# Create ~/bin
cd ~/
if dir ~/bin does not exist
then mkdir ~/bin
#!/bin/bash
# Create ~/blahhed && ~/blahs
if dir ~/blahhed does not exist
then mkdir ~/blahhed
if dir ~/blahs does not exist
then mkdir ~/blahs
# This will copy a file from ~/blahhed to ~/blahs
if ~/blahhed/file exists
then mv ~/blahhed/file ~/blahs/file
rm ~/blahhed/file
else loop for 60s
# This appends the date and time
# to the end of the file name
date_formatted=$(date +%m_%d_%y-%H,%M,%S)
if ~/blahs/file does exist
then mv ~/blahs/file ~/blahs/file.$date_formatted
rm ~/blahs/file
else loop for 60s
Ok Ive rewritten it like this am I on the right track here?
# Create ~/bin
cd ~/
if [! -d ~/bin]; then
mkdir ~/bin
if [ -d ~/bin]; then
#!/bin/bash
# Create ~/blahhed && ~/blahs
if [! -d ~/blahhed]; then
mkdir ~/blahhed
if [! -d ~/blahs]; then
mkdir ~/blahs
# This will copy a file from ~/blahhed to ~/blahs
while if [ -d ~/blahhed/file]; then
do
mv ~/blahhed/file ~/blahs/file
rm ~/blahhed/file
continue
# This appends the date and time
# to the end of the file name
date_formatted=$(date +%m_%d_%y-%H,%M,%S)
if [! -d ~/blahs/file]; then
mv ~/blahs/file ~/blahs/file.$date_formatted
rm ~/blahs/file
sleep 60 seconds
You could use watch(1) which is able to run a program or script every N seconds.
To run some script every few minutes (not seconds) - or every few hours or days, use some crontab(5) entries. To run it at some given (relative or absolute) time, consider at(1) (which you might use with some here document in your shell terminal, etc...).
However, to execute commands when a file exists or changes, you might use make(1) (which you could run from watch); that command is configurable in a Makefile (see documentation of GNU make)
And if you really care about file appearing or changing (and doing something on such changes), consider using inotify(7) based facilities, e.g. incrond with incrontab(5)
To test existence of directories or files, use test(1) often spelt as a [ , e.g.
## test in a script if directory ~/foo/ exist
if [ -d ~/foo/ ]; then
echo the directory foo exists
fi
Spaces are important above. You could use [ -d "$HOME/foo/" ]
It may look that you want to mimick logrotate(8). See also syslog(3) library function and logger(1) command.
To debug your bash script, start it (-see execve(2) & bash(1) for details- temporarily, while debugging) with
#!/bin/bash -vx
and make your foo.sh script executable with chmod a+x foo.sh
To stop execution of some script for some seconds, use sleep(1)
The mkdir(1) command accepts -p (and then won't create a directory if it already exists). mv(1) has also many options (including for backup).
To search some files in a file tree, use find(1). To search some content inside files, use grep. I also like ack
Read also Advanced Bash Scripting Guide & (if coding in C ...) Advanced Linux Programming and also the documentation of GNU bash (e.g. for shell builtins and control statements).
Did you consider using some revision control system like git ? It is useful to manage the evolution of source files (including shell scripts)
I've seen solutions similar to what you are asking, but using crontab with find -mmin 1 which will search for any files with a modtime <= 60 seconds within specified location.
Something along these lines (untested):
$ -> vi /tmp/file_finder.sh
# Add the following lines
#!/bin/bash
find /path/to/check -mmin 1 -type -f | while read fname; do
echo "$fname"
done
# Change perms
$ -> chmod 755 /tmp/file_finder.sh
$ -> crontab -e
* * * * * /tmp/file_finder.sh
With the above, you have now setup the cron to run every minute, and kick off a script that will search given directory for files with a modtime <= 60 seconds (new or updated).
Caveat: You should look for files with a mod time up to 5 minutes, that way you don't consider a file which may still be in the process of being written too.
I think you answered yourself (kind of)
Some suggestions:
1- use a while loop and at the end add sleep 60
2- write your procedure in a file (ex.; test1)
and then
watch -n 60 ./test1

Create a detailed self tracing log in bash

I know you can create a log of the output by typing in script nameOfLog.txt and exit in terminal before and after running the script, but I want to write it in the actual script so it creates a log automatically. There is a problem I'm having with the exec >>log_file 2>&1 line:
The code redirects the output to a log file and a user can no longer interact with it. How can I create a log where it just basically copies what is in the output?
And, is it possible to have it also automatically record the process of files that were copied? For example, if a file at /home/user/Deskop/file.sh was copied to /home/bckup, is it possible to have that printed in the log too or will I have to write that manually?
Is it also possible to record the amount of time it took to run the whole process and count the number of files and directories that were processed or am I going to have to write that manually too?
My future self appreciates all the help!
Here is my whole code:
#!/bin/bash
collect()
{
find "$directory" -name "*.sh" -print0 | xargs -0 cp -t ~/bckup #xargs handles files names with spaces. Also gives error of "cp: will not overwrite just-created" even if file didn't exist previously
}
echo "Starting log"
exec >>log_file 2>&1
timelimit=10
echo "Please enter the directory that you would like to collect.
If no input in 10 secs, default of /home will be selected"
read -t $timelimit directory
if [ ! -z "$directory" ] #if directory doesn't have a length of 0
then
echo -e "\nYou want to copy $directory." #-e is so the \n will work and it won't show up as part of the string
else
directory=/home/
echo "Time's up. Backup will be in $directory"
fi
if [ ! -d ~/bckup ]
then
echo "Directory does not exist, creating now"
mkdir ~/bckup
fi
collect
echo "Finished collecting"
exit 0
To answer the "how to just copy the output" question: use a program called tee and then a bit of exec magic explained here:
redirect COPY of stdout to log file from within bash script itself
Regarding the analytics (time needed, files accessed, etc) -- this is a bit harder. Some programs that can help you are time(1):
time - run programs and summarize system resource usage
and strace(1):
strace - trace system calls and signals
Check the man pages for more info. If you have control over the script it will be probably easier to do the logging yourself instead of parsing strace output.

shell script to create folder daily with time-stamp and push time-stamp generated logs

I have a cron job which runs every 30 minutes to generate log files with time-stamp like this:
test20130215100531.log,
test20130215102031.log
I would like to create one folder daily with date time-stamp and push log files in to respective date folder when generated.
I need to achieve this on AIX server with bash.
Maybe you are looking for a script like this:
#!/bin/bash
shopt -s nullglob # This line is so that it does not complain when no logfiles are found
for filename in test*.log; do # Files considered are the ones starting with test and ending in .log
foldername=$(echo "$filename" | awk '{print (substr($0, 5, 8));}'); # The foldername is characters 5 to 13 from the filename (if they exist)
mkdir -p "$foldername" # -p so that we don't get "folder exists" warning
mv "$filename" "$foldername"
echo "$filename $foldername" ;
done
I only tested with your sample, so do a proper testing before using in a directory that contains important stuff.
Edit in response to comments:
Change your original script to this:
foldername=$(date +%Y%m%d)
mkdir -p /home/app/logs/"$foldername"
sh sample.sh > /home/app/logs/"$foldername"/test$(date +%Y%m%d%H%M%S).log
Or if the directory is created somewhere else, just do this:
sh sample.sh > /home/app/logs/$(date +%Y%m%d)/test$(date +%Y%m%d%H%M%S).log
You should use logrotate! It can do this for you already, and you can just write to the same log file.
Check their man pages for info:
http://linuxcommand.org/man_pages/logrotate8.html

Resources