I have a cron job which runs every 30 minutes to generate log files with time-stamp like this:
test20130215100531.log,
test20130215102031.log
I would like to create one folder daily with date time-stamp and push log files in to respective date folder when generated.
I need to achieve this on AIX server with bash.
Maybe you are looking for a script like this:
#!/bin/bash
shopt -s nullglob # This line is so that it does not complain when no logfiles are found
for filename in test*.log; do # Files considered are the ones starting with test and ending in .log
foldername=$(echo "$filename" | awk '{print (substr($0, 5, 8));}'); # The foldername is characters 5 to 13 from the filename (if they exist)
mkdir -p "$foldername" # -p so that we don't get "folder exists" warning
mv "$filename" "$foldername"
echo "$filename $foldername" ;
done
I only tested with your sample, so do a proper testing before using in a directory that contains important stuff.
Edit in response to comments:
Change your original script to this:
foldername=$(date +%Y%m%d)
mkdir -p /home/app/logs/"$foldername"
sh sample.sh > /home/app/logs/"$foldername"/test$(date +%Y%m%d%H%M%S).log
Or if the directory is created somewhere else, just do this:
sh sample.sh > /home/app/logs/$(date +%Y%m%d)/test$(date +%Y%m%d%H%M%S).log
You should use logrotate! It can do this for you already, and you can just write to the same log file.
Check their man pages for info:
http://linuxcommand.org/man_pages/logrotate8.html
Related
By using bash script, I'm trying to detect whether a file has been created on a directory or not while running commands. Let me illustrate the problem;
#!/bin/bash
# give base directory to watch file changes
WATCH_DIR=./tmp
# get list of files on that directory
FILES_BEFORE= ls $WATCH_DIR
# actually a command is running here but lets assume I've created a new file there.
echo >$WATCH_DIR/filename
# and I'm getting new list of files.
FILES_AFTER= ls $WATCH_DIR
# detect changes and if any changes has been occurred exit the program.
After that I've just tried to compare these FILES_BEFORE and FILES_AFTER however couldn't accomplish that. I've tried;
comm -23 <($FILES_AFTER |sort) <($FILES_BEFORE|sort)
diff $FILES_AFTER $FILES_BEFORE > /dev/null 2>&1
cat $FILES_AFTER $FILES_BEFORE | sort | uniq -u
None of them gave me a result to understand there is a change or not. What I need is detecting the change and exiting the program if any. I am not really good at this bash script, searched a lot on the internet however couldn't find what I need. Any help will be appreciated. Thanks.
Thanks to informative comments, I've just realized that I've missed the basics of bash script but finally made that work. I'll leave my solution here as an answer for those who struggle like me.:
WATCH_DIR=./tmp
FILES_BEFORE=$(ls $WATCH_DIR)
echo >$WATCH_DIR/filename
FILES_AFTER=$(ls $WATCH_DIR)
if diff <(echo "$FILES_AFTER") <(echo "$FILES_BEFORE")
then
echo "No changes"
else
echo "Changes"
fi
It outputs "Changes" on the first run and "No Changes" for the other unless you delete the newly added documents.
I'm trying to interpret your script (which contains some errors) into an understanding of your requirements.
I think the simplest way is simply to rediect the ls command outputto named files then diff those files:
#!/bin/bash
# give base directory to watch file changes
WATCH_DIR=./tmp
# get list of files on that directory
ls $WATCH_DIR > /tmp/watch_dir.before
# actually a command is running here but lets assume I've created a new file there.
echo >$WATCH_DIR/filename
# and I'm getting new list of files.
ls $WATCH_DIR > /tmp/watch_dir.after
# detect changes and if any changes has been occurred exit the program.
diff -c /tmp/watch_dir.after /tmp/watch_dir.before
If the any files are modified by the 'commands', i.e. the files exists in the 'before' list, but might change, the above will not show that as a difference.
In this case you might be better off using a 'marker' file created to mark the instance the monitoring started, then use the find command to list any newer/modified files since the market file. Something like this:
#!/bin/bash
# give base directory to watch file changes
WATCH_DIR=./tmp
# get list of files on that directory
ls $WATCH_DIR > /tmp/watch_dir.before
# actually a command is running here but lets assume I've created a new file there.
echo >$WATCH_DIR/filename
# and I'm getting new list of files.
find $WATCH_DIR -type f -newer /tmp/watch_dir.before -exec ls -l {} \;
What this won't do is show any files that were deleted, so perhaps a hybrid list could be used.
Here is how I got it to work. It's also setup up so that you can have multiple watched directories with the same script with cron.
for example, if you wanted one to run every minute.
* * * * * /usr/local/bin/watchdir.sh /makepdf
and one every hour.
0 * * * * /user/local/bin/watchdir.sh /incoming
#!/bin/bash
WATCHDIR="$1"
NEWFILESNAME=.newfiles$(basename "$WATCHDIR")
if [ ! -f "$WATCHDIR"/.oldfiles ]
then
ls -A "$WATCHDIR" > "$WATCHDIR"/.oldfiles
fi
ls -A "$WATCHDIR" > $NEWFILESNAME
DIRDIFF=$(diff "$WATCHDIR"/.oldfiles $NEWFILESNAME | cut -f 2 -d "")
for file in $DIRDIFF
do
if [ -e "$WATCHDIR"/$file ];then
#do what you want to the file(s) here
echo $file
fi
done
rm $NEWFILESNAME
I am working on bash to create a back up system. My code is
#!/bin/bash
if [ ! -d "BackUp" ]
then
mkdir BackUp
fi
echo "enter number of access days you want to take for back up."
read days
bak="$(find . -mtime +$days)"
for file in $bak
do
mv $file BackUp
done
tar -cvf BackUp.tgz BackUp >> backUp.log
So, currently I am only taking log file from tar. so it does not prints the full path it only takes current working directory for text in log file.My last line of code takes up input for log file.
But the path stored is
.BackUp/foo1
.BackUp/foo2
.BackUp/foo3
instead i want it to be
home/ubuntu/Downloads/BackUp/foo1
home/ubuntu/Downloads/BackUp/foo2
home/ubuntu/Downloads/BackUp/foo3
You could store the absolute path in a variable and use it in the tar command:
BackUpDirFullPath=$(cd BackUp && pwd)
As command substitution invokes a subshell you are not leaving the current directory by executing cd.
Update:
In order to make -v output absolute paths (on Mac OS) I had to change to the root directory in a subshell and execute it from there ... something like that:
(cd / && tar -cvf /$OLDPWD/BackUp.tgz $BackUpDirFullPath)
This does output absolute paths ... in order to preserve the leading / you might try -P which preserves path names.
Basically I want to do something like this from bash.
if a file exists in a directory rename,move,whatever
if it doesn't exist loop every 60 seconds:
# Create ~/bin
cd ~/
if dir ~/bin does not exist
then mkdir ~/bin
#!/bin/bash
# Create ~/blahhed && ~/blahs
if dir ~/blahhed does not exist
then mkdir ~/blahhed
if dir ~/blahs does not exist
then mkdir ~/blahs
# This will copy a file from ~/blahhed to ~/blahs
if ~/blahhed/file exists
then mv ~/blahhed/file ~/blahs/file
rm ~/blahhed/file
else loop for 60s
# This appends the date and time
# to the end of the file name
date_formatted=$(date +%m_%d_%y-%H,%M,%S)
if ~/blahs/file does exist
then mv ~/blahs/file ~/blahs/file.$date_formatted
rm ~/blahs/file
else loop for 60s
Ok Ive rewritten it like this am I on the right track here?
# Create ~/bin
cd ~/
if [! -d ~/bin]; then
mkdir ~/bin
if [ -d ~/bin]; then
#!/bin/bash
# Create ~/blahhed && ~/blahs
if [! -d ~/blahhed]; then
mkdir ~/blahhed
if [! -d ~/blahs]; then
mkdir ~/blahs
# This will copy a file from ~/blahhed to ~/blahs
while if [ -d ~/blahhed/file]; then
do
mv ~/blahhed/file ~/blahs/file
rm ~/blahhed/file
continue
# This appends the date and time
# to the end of the file name
date_formatted=$(date +%m_%d_%y-%H,%M,%S)
if [! -d ~/blahs/file]; then
mv ~/blahs/file ~/blahs/file.$date_formatted
rm ~/blahs/file
sleep 60 seconds
You could use watch(1) which is able to run a program or script every N seconds.
To run some script every few minutes (not seconds) - or every few hours or days, use some crontab(5) entries. To run it at some given (relative or absolute) time, consider at(1) (which you might use with some here document in your shell terminal, etc...).
However, to execute commands when a file exists or changes, you might use make(1) (which you could run from watch); that command is configurable in a Makefile (see documentation of GNU make)
And if you really care about file appearing or changing (and doing something on such changes), consider using inotify(7) based facilities, e.g. incrond with incrontab(5)
To test existence of directories or files, use test(1) often spelt as a [ , e.g.
## test in a script if directory ~/foo/ exist
if [ -d ~/foo/ ]; then
echo the directory foo exists
fi
Spaces are important above. You could use [ -d "$HOME/foo/" ]
It may look that you want to mimick logrotate(8). See also syslog(3) library function and logger(1) command.
To debug your bash script, start it (-see execve(2) & bash(1) for details- temporarily, while debugging) with
#!/bin/bash -vx
and make your foo.sh script executable with chmod a+x foo.sh
To stop execution of some script for some seconds, use sleep(1)
The mkdir(1) command accepts -p (and then won't create a directory if it already exists). mv(1) has also many options (including for backup).
To search some files in a file tree, use find(1). To search some content inside files, use grep. I also like ack
Read also Advanced Bash Scripting Guide & (if coding in C ...) Advanced Linux Programming and also the documentation of GNU bash (e.g. for shell builtins and control statements).
Did you consider using some revision control system like git ? It is useful to manage the evolution of source files (including shell scripts)
I've seen solutions similar to what you are asking, but using crontab with find -mmin 1 which will search for any files with a modtime <= 60 seconds within specified location.
Something along these lines (untested):
$ -> vi /tmp/file_finder.sh
# Add the following lines
#!/bin/bash
find /path/to/check -mmin 1 -type -f | while read fname; do
echo "$fname"
done
# Change perms
$ -> chmod 755 /tmp/file_finder.sh
$ -> crontab -e
* * * * * /tmp/file_finder.sh
With the above, you have now setup the cron to run every minute, and kick off a script that will search given directory for files with a modtime <= 60 seconds (new or updated).
Caveat: You should look for files with a mod time up to 5 minutes, that way you don't consider a file which may still be in the process of being written too.
I think you answered yourself (kind of)
Some suggestions:
1- use a while loop and at the end add sleep 60
2- write your procedure in a file (ex.; test1)
and then
watch -n 60 ./test1
--Bash 4.1.17 (running with Cygwin)
Hello, I am trying to pass the date into the --suffix option on the move (mv) command. I am able to pass in a simple string (like my name) but unable to pass in the date. If you run the script below you will see that the mv command with the suffix="$var" works but suffix="$now" does not.
#!/bin/bash
dir="your directory goes here"
now="$(date "+%m/%d/%y")"
var="_CARL!!!"
echo "$now"
echo "$var"
cd "$dir"
touch test.txt
# error if already exists
mkdir ./stack_question
touch ./stack_question/test.txt
mv -b --suffix="$var" test.txt ./stack_question/
The idea is that if test.txt already exists when trying to move the file, the file will have a suffix appended to it. So if you run this script with:
--suffix="$var"
you will see that the stack_question directory contains two files:
test.txt & test.txt_CARL!!!
But, if you run this script with:
--suffix="$now"
you will see that in the stack_question directory only contains:
test.txt
Any help on this would be greatly appreciated!
It is because you have embedded / in your date format try
now="$(date +%m_%d_%y)"
I use a program to rip radio music. Sadly one can not set the temporary folder apart from the folder where the finished mp3's end up later. So I cannot set the output folder to auto add to iTunes.
I'm alright in coding java and what not but have no experience with shell scripts.
I need a script that iterates through all the files within a folder like every 10 minutes and moves them to a different location if they don't start with the string "Track". All temp files are called "Track..." so it should only move finished ones then. Could anyone give me a help getting started?
Thanks!
Here's an example script. You should set the DESTINATION directory properly before uncommenting the line which moves the files. Otherwise, you may end up moving them somewhere undesirable.
In the terminal, cd to the location where you save the snippet below and run the following commands to execute.
Prep work:
cd /save/location
chmod +x file_mover.sh # makes the file executable
Schedule a job:
crontab -e
*/10 * * * * /path/to/file_mover.sh
crontab -l # view list of scheduled jobs
With some minor tweaks you can make this accept CLI options.
#!/bin/bash
# files to skip
REGEX='^TRACK'
# location to move the files
DESTINATION=/tmp/mydir
# directory to read from
# PWD is the working directory
TARGET=${PWD}
# make the directory(ies) if it doesn't exists
if [ ! -f ${DESTINATION} ]; then
mkdir -p ${DESTINATION}
fi
# get the collection of files in the
for FILE in $( ls ${TARGET} )
do
# if the current file does not begin with TRACK, move it
if [[ ! ${FILE} =~ ${REGEX} ]]; then
echo ${FILE}
# SET THE DESTINATION DIRECTORY BEFORE UNCOMMENTING THE LINE BELOW
# if [ -f ${FILE} ]; then # uncomment if you want to
# ensure it's a file and not a directory
# mv ${FILE} ${DESTINATION} # move the file
# fi # uncomment to ensure it's a file (end if)
fi
done
Edit the crontab with EDITOR=nano crontab -e and add a line like this:
*/10 * * * * shopt -s extglob; mv ~/Music/Temp/!(Track)*.mp3 ~/Music/iTunes/iTunes\ Media/Automatically\ Add\ to\ iTunes.localized/
shopt -s extglob adds support for !(). See /usr/share/doc/bash/bash.html.