can Linux Bash search for file every 60 seconds and execute file commands? how would I do this? - bash

Basically I want to do something like this from bash.
if a file exists in a directory rename,move,whatever
if it doesn't exist loop every 60 seconds:
# Create ~/bin
cd ~/
if dir ~/bin does not exist
then mkdir ~/bin
#!/bin/bash
# Create ~/blahhed && ~/blahs
if dir ~/blahhed does not exist
then mkdir ~/blahhed
if dir ~/blahs does not exist
then mkdir ~/blahs
# This will copy a file from ~/blahhed to ~/blahs
if ~/blahhed/file exists
then mv ~/blahhed/file ~/blahs/file
rm ~/blahhed/file
else loop for 60s
# This appends the date and time
# to the end of the file name
date_formatted=$(date +%m_%d_%y-%H,%M,%S)
if ~/blahs/file does exist
then mv ~/blahs/file ~/blahs/file.$date_formatted
rm ~/blahs/file
else loop for 60s
Ok Ive rewritten it like this am I on the right track here?
# Create ~/bin
cd ~/
if [! -d ~/bin]; then
mkdir ~/bin
if [ -d ~/bin]; then
#!/bin/bash
# Create ~/blahhed && ~/blahs
if [! -d ~/blahhed]; then
mkdir ~/blahhed
if [! -d ~/blahs]; then
mkdir ~/blahs
# This will copy a file from ~/blahhed to ~/blahs
while if [ -d ~/blahhed/file]; then
do
mv ~/blahhed/file ~/blahs/file
rm ~/blahhed/file
continue
# This appends the date and time
# to the end of the file name
date_formatted=$(date +%m_%d_%y-%H,%M,%S)
if [! -d ~/blahs/file]; then
mv ~/blahs/file ~/blahs/file.$date_formatted
rm ~/blahs/file
sleep 60 seconds

You could use watch(1) which is able to run a program or script every N seconds.
To run some script every few minutes (not seconds) - or every few hours or days, use some crontab(5) entries. To run it at some given (relative or absolute) time, consider at(1) (which you might use with some here document in your shell terminal, etc...).
However, to execute commands when a file exists or changes, you might use make(1) (which you could run from watch); that command is configurable in a Makefile (see documentation of GNU make)
And if you really care about file appearing or changing (and doing something on such changes), consider using inotify(7) based facilities, e.g. incrond with incrontab(5)
To test existence of directories or files, use test(1) often spelt as a [ , e.g.
## test in a script if directory ~/foo/ exist
if [ -d ~/foo/ ]; then
echo the directory foo exists
fi
Spaces are important above. You could use [ -d "$HOME/foo/" ]
It may look that you want to mimick logrotate(8). See also syslog(3) library function and logger(1) command.
To debug your bash script, start it (-see execve(2) & bash(1) for details- temporarily, while debugging) with
#!/bin/bash -vx
and make your foo.sh script executable with chmod a+x foo.sh
To stop execution of some script for some seconds, use sleep(1)
The mkdir(1) command accepts -p (and then won't create a directory if it already exists). mv(1) has also many options (including for backup).
To search some files in a file tree, use find(1). To search some content inside files, use grep. I also like ack
Read also Advanced Bash Scripting Guide & (if coding in C ...) Advanced Linux Programming and also the documentation of GNU bash (e.g. for shell builtins and control statements).
Did you consider using some revision control system like git ? It is useful to manage the evolution of source files (including shell scripts)

I've seen solutions similar to what you are asking, but using crontab with find -mmin 1 which will search for any files with a modtime <= 60 seconds within specified location.
Something along these lines (untested):
$ -> vi /tmp/file_finder.sh
# Add the following lines
#!/bin/bash
find /path/to/check -mmin 1 -type -f | while read fname; do
echo "$fname"
done
# Change perms
$ -> chmod 755 /tmp/file_finder.sh
$ -> crontab -e
* * * * * /tmp/file_finder.sh
With the above, you have now setup the cron to run every minute, and kick off a script that will search given directory for files with a modtime <= 60 seconds (new or updated).
Caveat: You should look for files with a mod time up to 5 minutes, that way you don't consider a file which may still be in the process of being written too.

I think you answered yourself (kind of)
Some suggestions:
1- use a while loop and at the end add sleep 60
2- write your procedure in a file (ex.; test1)
and then
watch -n 60 ./test1

Related

mv command and rename not working on multiple flies

Below is a bash script to move files around and rename them. The problem is it doesn't work when there is more than one file in the directory. I'm assuming because the last parameter in the mv command is a file. Any suggestions?
'#!/bin/bash'
'INPUTDIR="/home/southern-uniontn/S001007420"'
'OUTPUTDIR="/mnt/edi-06/southern-uniontn/flats-in"'
'BACKUPDIR="/backup/southern-uniontn/S001007420"'
YEAR=`date +%Y`
MONTH=`date +%m`
DAY=`date +%d`
HOUR=`date +%H`
MINUTE=`date +%M`
######## Do some error checking #########
# Does backup dir exist?
if [ ! -d $BACKUPDIR/$YEAR ]
then
mkdir $BACKUPDIR/$YEAR
fi
if [ ! -d $BACKUPDIR/$YEAR/$MONTH ]
then
mkdir $BACKUPDIR/$YEAR/$MONTH
fi
if [ ! -d $BACKUPDIR/$YEAR/$MONTH/$DAY ]
then
mkdir $BACKUPDIR/$YEAR/$MONTH/$DAY
fi
if [[ $(find $INPUTDIR -type f | wc -l) -gt 0 ]];
then
###### Rename the file, move it to Backup, then copy to the Output Directory #####
for f in $INPUTDIR/*
do
echo "`date` - Move recurring txt flat file to BackupDir for Union TN from Southern"
mv $INPUTDIR/* $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt
sleep 2
echo "`date` - Copy backup file to the Union TN Output Directory"
cp $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt $OUTPUTDIR/
done;
fi
Some notes:
Get out of the habit of using ALLCAPS variable names, leave those as reserved
by the shell. One day you'll write PATH=something and then wonder
why your script is
broken.
mkdir -p can create parent directories, and will not error if the dir already exists
store the filenames in an array. Then the shell does not have to duplicate
the work, and you don't need to count how many there are: if there are no
files, the loop has zero iterations
if you want to keep the same directory hierarchy in the outputdir,
you need to do that by hand.
use read to get the date parts
with bash v4.2+, printf can be used instead of calling out to date
use magic value "-1" to mean "now".
printf '%(%Y-%m-%d)T\n' -1 prints "2021-10-25" (as of the day I write this)
This is, I think, what you want:
#!/bin/bash
inputdir='/home/southern-uniontn/S001007420'
outputdir='/mnt/edi-06/southern-uniontn/flats-in'
backupdir='/backup/southern-uniontn/S001007420'
read year month day hour minute < <(printf '%(%Y %m %d %H %M)T\n' -1)
# create backup dirs if not exists
date_dir="$year/$month/$day"
mkdir -p "$backupdir/$date_dir"
mkdir -p "$outputdir/$date_dir"
mapfile -t files < <(find $inputdir -type f)
for f in "${files[#]}"
do
###### Rename the file, move it to Backup, then copy to the Output Directory #####
backup_file="UnionTN-S001007420-$year$month$day-$hour$minute.txt"
printf '%(%c)T - Move recurring txt flat file to backupdir for Union TN from Southern\n' -1
mv "$f" "$backupdir/$date_dir/$backup_file"
printf '%(%c)T - Copy backup file to the Union TN Output Directory\n' -1
cp "$backupdir/$date_dir/$backup_file" "$outputdir/$date_dir/$backup_file"
done
When using a glob with mv, the target must be an existing directory, and all matching files will be moved inside that directory.
In your case,
mv $INPUTDIR/* $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt
tells mv to move all file inside the $INPUTDIR/* directory to a directory named $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt.
I'm not sure what you're trying to do, but I hope this help.
Some more advice you could use:
Don't put the shebang (the first line beginning with "#") and the first three variable declarations inside single-quotes.
Some argue it is more portable and better to write /usr/bin/env bash instead of /bin/bash in the shebang
if [ CONDITION ] /then ACTION /fi statements can be simplified by writing [ CONDITION ] && ACTION
You reduce your likely hood of encountering unexpected behaviour when double-quoting your strings and variable (i.e. write "${year}/${month}/" instead of $year/$month.
No need to call mkdir a, followed by mkidr a/b, then mkdir a/b/c and so on, you can just call mkdir -p a/b/c. The p flag tells mkdir to create parent directories if they don't already exist.
It is unnecessary to validate the existence of a directory before calling mkdir since mkdir already validates that for you.
As pointed out by commenters, all-caps variables are conventions for special POSIX related variables. You should use another type of casing.
You could use date to do the formatting for you: date +%Y/%m/%d will print 2021/10/25
Strings without interpolation can have single-quotes.
(Optional, prevent undesired behaviors) Put set -e at the beginning of your scripts, after the shebang, to tell bash to halt if an error is encountered
And finally, use man <command_name> for built-in documentation!

Bash script to check if a new file has been created on a directory after run a command

By using bash script, I'm trying to detect whether a file has been created on a directory or not while running commands. Let me illustrate the problem;
#!/bin/bash
# give base directory to watch file changes
WATCH_DIR=./tmp
# get list of files on that directory
FILES_BEFORE= ls $WATCH_DIR
# actually a command is running here but lets assume I've created a new file there.
echo >$WATCH_DIR/filename
# and I'm getting new list of files.
FILES_AFTER= ls $WATCH_DIR
# detect changes and if any changes has been occurred exit the program.
After that I've just tried to compare these FILES_BEFORE and FILES_AFTER however couldn't accomplish that. I've tried;
comm -23 <($FILES_AFTER |sort) <($FILES_BEFORE|sort)
diff $FILES_AFTER $FILES_BEFORE > /dev/null 2>&1
cat $FILES_AFTER $FILES_BEFORE | sort | uniq -u
None of them gave me a result to understand there is a change or not. What I need is detecting the change and exiting the program if any. I am not really good at this bash script, searched a lot on the internet however couldn't find what I need. Any help will be appreciated. Thanks.
Thanks to informative comments, I've just realized that I've missed the basics of bash script but finally made that work. I'll leave my solution here as an answer for those who struggle like me.:
WATCH_DIR=./tmp
FILES_BEFORE=$(ls $WATCH_DIR)
echo >$WATCH_DIR/filename
FILES_AFTER=$(ls $WATCH_DIR)
if diff <(echo "$FILES_AFTER") <(echo "$FILES_BEFORE")
then
echo "No changes"
else
echo "Changes"
fi
It outputs "Changes" on the first run and "No Changes" for the other unless you delete the newly added documents.
I'm trying to interpret your script (which contains some errors) into an understanding of your requirements.
I think the simplest way is simply to rediect the ls command outputto named files then diff those files:
#!/bin/bash
# give base directory to watch file changes
WATCH_DIR=./tmp
# get list of files on that directory
ls $WATCH_DIR > /tmp/watch_dir.before
# actually a command is running here but lets assume I've created a new file there.
echo >$WATCH_DIR/filename
# and I'm getting new list of files.
ls $WATCH_DIR > /tmp/watch_dir.after
# detect changes and if any changes has been occurred exit the program.
diff -c /tmp/watch_dir.after /tmp/watch_dir.before
If the any files are modified by the 'commands', i.e. the files exists in the 'before' list, but might change, the above will not show that as a difference.
In this case you might be better off using a 'marker' file created to mark the instance the monitoring started, then use the find command to list any newer/modified files since the market file. Something like this:
#!/bin/bash
# give base directory to watch file changes
WATCH_DIR=./tmp
# get list of files on that directory
ls $WATCH_DIR > /tmp/watch_dir.before
# actually a command is running here but lets assume I've created a new file there.
echo >$WATCH_DIR/filename
# and I'm getting new list of files.
find $WATCH_DIR -type f -newer /tmp/watch_dir.before -exec ls -l {} \;
What this won't do is show any files that were deleted, so perhaps a hybrid list could be used.
Here is how I got it to work. It's also setup up so that you can have multiple watched directories with the same script with cron.
for example, if you wanted one to run every minute.
* * * * * /usr/local/bin/watchdir.sh /makepdf
and one every hour.
0 * * * * /user/local/bin/watchdir.sh /incoming
#!/bin/bash
WATCHDIR="$1"
NEWFILESNAME=.newfiles$(basename "$WATCHDIR")
if [ ! -f "$WATCHDIR"/.oldfiles ]
then
ls -A "$WATCHDIR" > "$WATCHDIR"/.oldfiles
fi
ls -A "$WATCHDIR" > $NEWFILESNAME
DIRDIFF=$(diff "$WATCHDIR"/.oldfiles $NEWFILESNAME | cut -f 2 -d "")
for file in $DIRDIFF
do
if [ -e "$WATCHDIR"/$file ];then
#do what you want to the file(s) here
echo $file
fi
done
rm $NEWFILESNAME

gnu parallel to parallelize a for loop

I have seen several questions about this topic, but I lack the ability to translate this to my specific problem. I have a for loop that loops through sub directories and then executes a .sh script on a compressed text file inside each directory. I want to parallelize this process, but I'm struggling to apply gnu parallel.
Here is my loop:
for d in ./*/ ; do (cd "$d" && script.sh); done
I understand I need to input a list into parallel, so i have been trying this:
ls -d */ | parallel cd && script.sh
While this appears to get started, I get an error when gzip tries to unzip one of the txt files inside the directory, saying the file does not exist:
gzip: *.txt.gz: No such file or directory
However, when I run the original for loop, I have no issues aside from it taking a century to finish. Also, I only get the gzip error once when using parallel, which is so weird considering I have over 1000 sub-directories.
My questions are:
How do I get Parallel to work in my case? How do I get parallel to parallelize the application of a .sh script to 1000s of files in their own sub-directories? ie- what is the solution to my problem? I gotta make progress.
What am I missing? Syntax, loop, bad script? I want to learn.
Is Parallel actually attempting to run all these .sh scripts in parallel? Why dont I get an error for every .txt.gz file?
Is parallel the best option for the application? Is there another option that is better suited to my needs?
Two problems:
In:
ls -d */ | parallel cd && script.sh
what is paralleled is just cd, not script.sh. script.sh is only executed once, after all parallel cd jobs have run, if there was no error. It is the same as:
ls -d */ | parallel cd
if [ $? -eq 0 ]; then script.sh; fi
You do not pass the target directory to cd. So, what is executed by parallel is just cd, which just changes the current directory to your home directory. The final script.sh is executed in the current directory (from where you invoked the command) where there are probably no *.txt.gz files, thus the error.
You can check yourself the effect of the first problem with:
$ mkdir /tmp/foobar && cd /tmp/foobar && mkdir a b c
$ ls -d */ | parallel cd && pwd
/tmp/foobar
The output of pwd is printed only once, even if you have more than one input directory. You can fix it by quoting the command and then check the second problem with:
$ ls -d */ | parallel 'cd && pwd'
/homes/myself
/homes/myself
/homes/myself
You should see as many pwd outputs as there are input directories but it is always the same output: your home directory. You can fix the second problem by using the {} replacement string that is substituted with the current input. Check it with:
$ ls -d */ | parallel 'cd {} && pwd'
/tmp/foobar/a
/tmp/foobar/b
/tmp/foobar/c
Now, you should have all input directories properly listed in the output.
For your specific problem this should work:
ls -d */ | parallel 'cd {} && script.sh'

shell script to create folder daily with time-stamp and push time-stamp generated logs

I have a cron job which runs every 30 minutes to generate log files with time-stamp like this:
test20130215100531.log,
test20130215102031.log
I would like to create one folder daily with date time-stamp and push log files in to respective date folder when generated.
I need to achieve this on AIX server with bash.
Maybe you are looking for a script like this:
#!/bin/bash
shopt -s nullglob # This line is so that it does not complain when no logfiles are found
for filename in test*.log; do # Files considered are the ones starting with test and ending in .log
foldername=$(echo "$filename" | awk '{print (substr($0, 5, 8));}'); # The foldername is characters 5 to 13 from the filename (if they exist)
mkdir -p "$foldername" # -p so that we don't get "folder exists" warning
mv "$filename" "$foldername"
echo "$filename $foldername" ;
done
I only tested with your sample, so do a proper testing before using in a directory that contains important stuff.
Edit in response to comments:
Change your original script to this:
foldername=$(date +%Y%m%d)
mkdir -p /home/app/logs/"$foldername"
sh sample.sh > /home/app/logs/"$foldername"/test$(date +%Y%m%d%H%M%S).log
Or if the directory is created somewhere else, just do this:
sh sample.sh > /home/app/logs/$(date +%Y%m%d)/test$(date +%Y%m%d%H%M%S).log
You should use logrotate! It can do this for you already, and you can just write to the same log file.
Check their man pages for info:
http://linuxcommand.org/man_pages/logrotate8.html

Mac Terminal search and move files not containing pattern in name

I use a program to rip radio music. Sadly one can not set the temporary folder apart from the folder where the finished mp3's end up later. So I cannot set the output folder to auto add to iTunes.
I'm alright in coding java and what not but have no experience with shell scripts.
I need a script that iterates through all the files within a folder like every 10 minutes and moves them to a different location if they don't start with the string "Track". All temp files are called "Track..." so it should only move finished ones then. Could anyone give me a help getting started?
Thanks!
Here's an example script. You should set the DESTINATION directory properly before uncommenting the line which moves the files. Otherwise, you may end up moving them somewhere undesirable.
In the terminal, cd to the location where you save the snippet below and run the following commands to execute.
Prep work:
cd /save/location
chmod +x file_mover.sh # makes the file executable
Schedule a job:
crontab -e
*/10 * * * * /path/to/file_mover.sh
crontab -l # view list of scheduled jobs
With some minor tweaks you can make this accept CLI options.
#!/bin/bash
# files to skip
REGEX='^TRACK'
# location to move the files
DESTINATION=/tmp/mydir
# directory to read from
# PWD is the working directory
TARGET=${PWD}
# make the directory(ies) if it doesn't exists
if [ ! -f ${DESTINATION} ]; then
mkdir -p ${DESTINATION}
fi
# get the collection of files in the
for FILE in $( ls ${TARGET} )
do
# if the current file does not begin with TRACK, move it
if [[ ! ${FILE} =~ ${REGEX} ]]; then
echo ${FILE}
# SET THE DESTINATION DIRECTORY BEFORE UNCOMMENTING THE LINE BELOW
# if [ -f ${FILE} ]; then # uncomment if you want to
# ensure it's a file and not a directory
# mv ${FILE} ${DESTINATION} # move the file
# fi # uncomment to ensure it's a file (end if)
fi
done
Edit the crontab with EDITOR=nano crontab -e and add a line like this:
*/10 * * * * shopt -s extglob; mv ~/Music/Temp/!(Track)*.mp3 ~/Music/iTunes/iTunes\ Media/Automatically\ Add\ to\ iTunes.localized/
shopt -s extglob adds support for !(). See /usr/share/doc/bash/bash.html.

Resources