Bash script to backup multiple directories specified in config file - bash

I'm having some difficulties with this. Basically, for work I need a bash script that backs up a variable number of directories that are stored in a config file.
I'm sure I need to import the list from the config file and just use a loop to copy all the directories across. I have it working for a single directory. My code is below. I've cut it down to a minimum.
#!/bin/sh
if [ ! -f ./backup.conf ]
then
echo "Configuration file not found. Exiting!!"
exit
fi
. ./backup.conf
unset PATH
# make sure we're running as root
if (( `$ID -u` != 0 )) ; then { $ECHO "Sorry, must be root. Exiting..."; exit; } fi ;
# attempt to remount the RW mount point as RW; else abort
$MOUNT -o remount,rw $SOURCEFILE $DESTINATIONFOLDER ;
if (( $? )); then
{
$ECHO "snapshot: could not remount $DESTINATIONFOLDER readwrite";
exit;
}
fi ;
# step 2: create new backup folder:
$MKDIR $FULLPATH
**Loop should go here**
#copy source directories to backup folder
$RSYNC \
-va --delete --delete-excluded \
--exclude-from="$EXCLUDES" \
$SOURCEFILE $FULLPATH;
The config file is as follows
SOURCE=path
DESTINATION=path2
BACKUPFOLDERNAME=/laptopBackup
My question is what is the best approach to do this task. i.e how should I format the config file to import a variable amount of paths to an array? or is there a better way of doing this?

I'd personally do it slightly differently and have my configuration file more of a "control file". For example:
/path /path2 /laptopBackup
/tmp /test /bigmachine
etc.. 1 line per mount, 3 fields per line (source, destination, backupfoldername)
Then use something like :
while read SOURCE DESTINATION BACKUPFOLDERNAME
do
<stuff>
done < ${configfile}
(removed the cat so as not to shame myself further :( )

Related

BASH Script for creating multiple directories, moving files, and then renaming said files

I am trying to make a bash script to create directories with the same name as each file in a given directory, then move said files to their respective directories, and then rename the files.
Basically - a quantum chemistry program that I use requires that the input files be named "ZMAT". So, if I have multiple jobs, I currently need to manually create directories, and then move the ZMAT files into them (can only run one job per folder).
When I run my code, I get "binary operator expected". I am not sure what this means. Some help please.
Here is what I have so far:
#!/bin/bash
if [ -e *.ZMAT ];
then
echo "CFOUR Job Detected"
for INPFILE in *.ZMAT; do
BASENAME=$(basename $INPFILE )
INPFILE=$BASENAME.ZMAT
OUTFILE=$BASENAME.out
XYZFILE=$BASENAME.xyz
ERRORFILE=$BASENAME.slu
if [ ! -e $ERRORFILE ];
then
# Create folder in scratch directory with the basename
mkdir /scratch/CFOUR/$BASENAME
# Move the file to its directory
mv -f $INPFILE /scratch/CFOUR/$BASENAME
# cd to the new directory
cd /scratch/CFOUR/$BASENAME
# Change the file name to just ZMAT
mv -f $INPFILE ZMAT
echo "Submitting CFOUR Job"
# Submit to scheduler
#RUN_COMMAND="sbatch -J $BASENAME _CFOUR_MRCC_SLURM.SUB"
#eval $RUN_COMMAND
else
echo "Error File Detected - Not Submitting Job"
fi
done
fi
An alternative would be to create symlinks to the original files.
As you said before, each ZMAT symlink would need to be in its own directory.
The upside is that the original data doesn't move, so less risk of breaking it, but the tool you want to use should read the symlinks as if they are the files it is looking for.
This one-liner creates an out directory in the current folder that you could subsequently move wherever you want it. You could easily create it where you do want it by replacing "out" with whatever absolute path you wanted
for i in *.ZMAT; do mkdir -p out/$i ; ln -s $PWD/$i out/$i/ZMAT ; done
I believe I have solved my problem. Here is the new script, which appears to be working fine. Any input is welcome though!
#!/bin/bash
SUBDIR=$(pwd)
for i in *.ZMAT; do
BASENAME=$(basename $i .ZMAT)
INPFILE=$BASENAME.ZMAT
OUTFILE=$BASENAME.out
XYZFILE=$BASENAME.xyz
ERRORFILE=$BASENAME.slu
if [ ! -e $ERRORFILE ];
then
mkdir /scratch/CFOUR/$BASENAME # Create Scratch Folder
cp $INPFILE /scratch/cdc/CFOUR/$BASENAME # Move Input to Scratch
cd /scratch/CFOUR/$BASENAME #cd to Scratch Folder
mv -f $INPFILE ZMAT # Change Input Name
echo "Submitting CFOUR Job"
# Submit to scheduler
#RUN_COMMAND="sbatch -J $BASENAME _CFOUR_MRCC_SLURM.SUB"
#eval $RUN_COMMAND
cd $SUBDIR #Go back to SUBDIR
else
echo "Error File Already Exists"
fi
done

bash check for subdirectories under directory

This is my first day scripting, I use linux but needed a script that I have been racking my brain until i finally ask for help. I need to check a directory that has directories already present to see if any new directories are added that are not expected.
Ok I think i have got this as simple as possible. The below works but displays all files in the directory as well. I will keep working at it unless someone can tell me how not to list the files too | I tried ls -d but it is doing the echo "nothing new". I feel like an idiot and should have got this sooner.
#!/bin/bash
workingdirs=`ls ~/ | grep -viE "temp1|temp2|temp3"`
if [ -d "$workingdirs" ]
then
echo "nothing new"
else
echo "The following Direcetories are now present"
echo ""
echo "$workingdirs"
fi
If you want to take some action when a new directory is created, used inotifywait. If you just want to check to see that the directories that exist are the ones you expect, you could do something like:
trap 'rm -f $TMPDIR/manifest' 0
# Create the expected values. Really, you should hand edit
# the manifest, but this is just for demonstration.
find "$Workingdir" -maxdepth 1 -type d > $TMPDIR/manifest
while true; do
sleep 60 # Check every 60 seconds. Modify period as needed, or
# (recommended) use inotifywait
if ! find "$Workingdir" -maxdepth 1 -type d | cmp - $TMPDIR/manifest; then
: Unexpected directories exist or have been removed
fi
done
Below shell script will show directory present or not.
#!/bin/bash
Workingdir=/root/working/
knowndir1=/root/working/temp1
knowndir2=/root/working/temp2
knowndir3=/root/working/temp3
my=/home/learning/perl
arr=($Workingdir $knowndir1 $knowndir2 $knowndir3 $my) #creating an array
for i in ${arr[#]} #checking for each element in array
do
if [ -d $i ]
then
echo "directory $i present"
else
echo "directory $i not present"
fi
done
output:
directory /root/working/ not present
directory /root/working/temp1 not present
directory /root/working/temp2 not present
directory /root/working/temp3 not present
**directory /home/learning/perl present**
This will save the available directories in a list to a file. When you run the script a second time, it will report directories that have been deleted or added.
#!/bin/sh
dirlist="$HOME/dirlist" # dir list file for saving state between runs
topdir='/some/path' # the directory you want to keep track of
tmpfile=$(mktemp)
find "$topdir" -type d -print | sort -o "$tmpfile"
if [ -f "$dirlist" ] && ! cmp -s "$dirlist" "$tmpfile"; then
echo 'Directories added:'
comm -1 -3 "$dirlist" "$tmpfile"
echo 'Directories removed:'
comm -2 -3 "$dirlist" "$tmpfile"
else
echo 'No changes'
fi
mv "$tmpfile" "$dirlist"
The script will have problems with directories that have very exotic names (containing newlines).

shell backup script renaming

I was able to script the backup process, but I want to make an another script for my storage server for a basic file rotation.
What I want to make:
I want to store my files in my /home/user/backup folder. Only want to store the 10 most fresh backup files and name them like this:
site_foo_date_1.tar site_foo_date_2.tar ... site_foo_date_10.tar
site_foo_date_1.tar being the most recent backup file.
Past num10 the file will be deleted.
My incoming files from the other server are simply named like this: site_foo_date.tar
How can I do this?
I tried:
DATE=`date "+%Y%m%d"`
cd /home/user/backup/com
if [ -f site_com_*_10.tar ]
then
rm site_com_*_10.tar
fi
FILES=$(ls)
for file in $FILES
do
echo "$file"
if [ "$file" != "site_com_${DATE}.tar" ]
then
str_new=${file:18:1}
new_str=$((str_new + 1))
to_rename=${file::18}
mv "${file}" "$to_rename$new_str.tar"
fi
done
file=$(ls | grep site_com_${DATE}.tar)
filename=`echo "$file" | cut -d'.' -f1`
mv "${file}" "${filename}_1.tar"
The main issue with your code is that looping through all files in the directory with ls * without some sort of filter is a dangerous thing to do.
Instead, I've used for i in $(seq 9 -1 1) to loop through files from *_9 to *_1 to move them. This ensures we only move backup files, and nothing else that may have accidentally got into the backup directory.
Additionally, relying on the sequence number to be the 18th character in the filename is also destined to break. What happens if you want more than 10 backups in the future? With this design, you can change 9 to be any number you like, even if it's more than 2 digits.
Finally, I added a check before moving site_com_${DATE}.tar in case it doesn't exist.
#!/bin/bash
DATE=`date "+%Y%m%d"`
cd "/home/user/backup/com"
if [ -f "site_com_*_10.tar" ]
then
rm "site_com_*_10.tar"
fi
# Instead of wildcarding all files in the directory
# this method picks out only the expected files so non-backup
# files are not changed. The renumbering is also made easier
# this way.
# Loop through from 9 to 1 in descending order otherwise
# the same file will be moved on each iteration
for i in $(seq 9 -1 1)
do
# Find and expand the requested file
file=$(find . -maxdepth 1 -name "site_com_*_${i}.tar")
if [ -f "$file" ]
then
echo "$file"
# Create new file name
new_str=$((i + 1))
to_rename=${file%_${i}.tar}
mv "${file}" "${to_rename}_${new_str}.tar"
fi
done
# Check for latest backup file
# and only move it if it exists.
file=site_com_${DATE}.tar
if [ -f $file ]
then
filename=${file%.tar}
mv "${file}" "${filename}_1.tar"
fi

Linux shell script to copy and rename multiple files

I have this snippet:
#!/bin/bash
parent=/parent
newfolder=/newfolder
mkdir "$newfolder"
for folder in "$parent"/*; do
if [[ -d $folder ]]; then
foldername="${folder##*/}"
for file in "$parent"/"$foldername"/*; do
filename="${file##*/}"
newfilename="$foldername"_"$filename"
cp "$file" "$newfolder"/"$newfilename"
done
fi
done
I do need to turn it around in a way that the copied files would be named after the folder they are being moved to (e.g. moving to the /root/Case22 files would be renamed to case22_1.jpg, case22_2.docx, case22_3.JPG etc). The files would be copied from USB and both destination and source directries would be entered by the user. I have written everything else and it works apart from actual renaming and thought I could adapt this snippet.
thanks
p.s. the snippet is written by Jahid and found on stackoverflow
you can try something like this;
#!/bin/bash
parent=/root
a=1
for file in $parent/Case22*; do
filename="${file%.*}"
extension="${file##*.}"
newfilename=$(printf "$filename"_"$a"."$extension")
mv -- "$file" "$newfilename"
let a=a+1
done
Thanks for the help. I have found the solution and thought I might post it here in case someone else will be looking at this.
As the title suggests I needed a Linux shell script to copy and rename multiple files keeping original directory tree (the file source and archive locations would be specified by the user of the script). Here is the code that I came up with after few days research of different sources (it includes a trap so only one instance of script would be running at a time):
lockdir=/var/tmp/mylock #directory for lock file creation
pidfile=/var/tmp/mylock/pid #directory to get the process ID number
if ( mkdir ${lockdir} ) 2> /dev/null; then #main argument to create lock file
echo $$ > $pidfile #if successful script will proceed, otherwise it will skip to the else part of the statement at the end of the script
trap 'rm -rf "$lockdir"; exit $?' INT TERM EXIT #trap to capture reasons of script termination and removal of the lock file so it could be launched again
#start of the main script body, part of successful "if" statement
# read entry_for_testing_only #request for user entry to keep script running and try to run another script instance
findir="$2/$(basename "$1")" #variable that defines final directory to which files from USB will be copied
if [ ! -d "$1" ]; then #testing if first directory entry is a valid directory’’
echo "$1" "is not a directory"
echo ""
exit
else
if [ ! -d "$2" ]; then #testing if second entry is a valid directory
echo "archive directory non existant"
exit
else
if [ -d "$findir" ] && [ "$(ls -A "$findir")" ]; then #testing if second entry directory contains the same name folders and if the folders are empty - to avoid file overwriting
echo "such folder already there and it's not empty"
exit
else
if [ ! -d "$findir" ] ; then #last archive directory argument to create final archive directory
mkdir "$findir"
else true
fi
fi
fi
fi
rsync -a "$1"/ "$findir" #command to copy all files from the source to the archive retaining the directory tree
moved_files="$(find "$findir" -type f)" #variable that finds all files that have been copied to the archive directory
for file in $moved_files; do #start of the loop that renames copied files
counter="$((counter+1))" #incrementation variable
source_name="$(basename "$1")" #variable that captures the name of the source directory
new_name="$source_name"_"$counter" #variable that puts start of the file name and incrementation element together
if echo "$file" | grep "\." #argument that captures the extension of the file
then
extension="$(echo "$file" | cut -f2 -d. )"
else
extension=
fi
full_name="$new_name"."$extension" #variable that defines the final new name of the file
dir="$(dirname "${file}")" #variable that captures the directorry address of currently looped file
mv "$file" "$dir/$full_name" #move command to rename currently looped file with the final new name
done
#end of the main script body, unsuccessful "if" statement continues here
else
echo "Another instance of this script is already running. PID: $(cat $pidfile)"
fi

Bash: Creating subdirectories reading from a file

I have a file that contains some keywords and I intend to create subdirectories into the same directory of the same keyword using a bash script. Here is the code I am using but it doesn't seem to be working.
I don't know where I have gone wrong. Help me out
for i in `cat file.txt`
do
# if [[ ! -e $path/$i ]]; then
echo "creating" $i "directory"
mkdir $path/$i
# fi
grep $i file >> $path/$i/output.txt
done
echo "created the files in "$path/$TEMP/output.txt
You've gone wrong here, and you've gone wrong here.
while read i
do
echo "Creating $i directory"
mkdir "$path/$i"
grep "$i" file >> "$path/$i"/output.txt
done < file.txt
echo "created the files in $path/$TEMP/output.txt"
78mkdir will refuse to create a directory, if parts of it do not exist.
e.g. if there is no /foo/bar directory, then mkdir /foo/bar/baz will fail.
you can relax this a bit by using the -p flag, which will create parent directories if necessary (in the example, it might create /foo and /foo/bar).
you should also use quotes, in case your paths contain blanks.
mkdir -p "${path}/${i}"
finally, make sure that you are actually allowed to create directories in $path

Resources