I'm creating a custom script to run backups using Clonezilla. The script is working as expected (so far) but I'm having a little trouble with the variable assignment.
The variable assignment line (maxBackups=4) creates a file called '4' in the current directory and the if test doesn't work properly.
(the way I understand it, the link to the parent directory counts as a directory with this directory counting method.)
What am I doing wrong? I know it is something simple...
Thanks
#!/bin/bash
# Automated usb backup script for Clonezilla
# by DRC
# Begin script
# Store date and time for use in folder name
# to a variable called folderName
folderName=$(date +"%Y-%m-%d--%H-%M")
# mount second partition of USB media for storing
# saved image
mount /dev/sdb2 /home/partimag/
# Determine if there are more than 3 directories
# and terminate the script if there are
cd /home/partimag
maxBackups=4
numberOfBackups=$(find -maxdepth 1 -type d | wc -l)
if [ $numberOfBackups > $maxBackups ]; then
echo "The maximum number of backups has been reached."
echo "Plese burn the backups to DVD and remove them"
echo "from the USB key."
echo "Press ENTER to continue and select 'Poweroff'"
echo "from the next menu."
# Wait for the user to press the enter key
read
# If there are three or less backups, a new backup will be made
else
/usr/sbin/ocs-sr -q2 -c -j2 -a -z0 -i 2000 -sc -p true savedisk $folderName sda
fi
maxBackups=4 is not creating a file named 4 in your directory. What is creating that file is the if [ $numberOfBackups > $maxBackups ] bit. > is redirection to an output file, since [ is a command, not a keyword. You could try one of these instead:
if [ $numberOfBackups -gt $maxBackups ] # -gt is test's version of greater than
if [[ $numberOfBackups > $maxBackups ]] # double brackets are keywords, not commands
if (( numberOfBackups > maxBackups )) # arithmetic context doesn't even require $
Related
Below is a bash script to move files around and rename them. The problem is it doesn't work when there is more than one file in the directory. I'm assuming because the last parameter in the mv command is a file. Any suggestions?
'#!/bin/bash'
'INPUTDIR="/home/southern-uniontn/S001007420"'
'OUTPUTDIR="/mnt/edi-06/southern-uniontn/flats-in"'
'BACKUPDIR="/backup/southern-uniontn/S001007420"'
YEAR=`date +%Y`
MONTH=`date +%m`
DAY=`date +%d`
HOUR=`date +%H`
MINUTE=`date +%M`
######## Do some error checking #########
# Does backup dir exist?
if [ ! -d $BACKUPDIR/$YEAR ]
then
mkdir $BACKUPDIR/$YEAR
fi
if [ ! -d $BACKUPDIR/$YEAR/$MONTH ]
then
mkdir $BACKUPDIR/$YEAR/$MONTH
fi
if [ ! -d $BACKUPDIR/$YEAR/$MONTH/$DAY ]
then
mkdir $BACKUPDIR/$YEAR/$MONTH/$DAY
fi
if [[ $(find $INPUTDIR -type f | wc -l) -gt 0 ]];
then
###### Rename the file, move it to Backup, then copy to the Output Directory #####
for f in $INPUTDIR/*
do
echo "`date` - Move recurring txt flat file to BackupDir for Union TN from Southern"
mv $INPUTDIR/* $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt
sleep 2
echo "`date` - Copy backup file to the Union TN Output Directory"
cp $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt $OUTPUTDIR/
done;
fi
Some notes:
Get out of the habit of using ALLCAPS variable names, leave those as reserved
by the shell. One day you'll write PATH=something and then wonder
why your script is
broken.
mkdir -p can create parent directories, and will not error if the dir already exists
store the filenames in an array. Then the shell does not have to duplicate
the work, and you don't need to count how many there are: if there are no
files, the loop has zero iterations
if you want to keep the same directory hierarchy in the outputdir,
you need to do that by hand.
use read to get the date parts
with bash v4.2+, printf can be used instead of calling out to date
use magic value "-1" to mean "now".
printf '%(%Y-%m-%d)T\n' -1 prints "2021-10-25" (as of the day I write this)
This is, I think, what you want:
#!/bin/bash
inputdir='/home/southern-uniontn/S001007420'
outputdir='/mnt/edi-06/southern-uniontn/flats-in'
backupdir='/backup/southern-uniontn/S001007420'
read year month day hour minute < <(printf '%(%Y %m %d %H %M)T\n' -1)
# create backup dirs if not exists
date_dir="$year/$month/$day"
mkdir -p "$backupdir/$date_dir"
mkdir -p "$outputdir/$date_dir"
mapfile -t files < <(find $inputdir -type f)
for f in "${files[#]}"
do
###### Rename the file, move it to Backup, then copy to the Output Directory #####
backup_file="UnionTN-S001007420-$year$month$day-$hour$minute.txt"
printf '%(%c)T - Move recurring txt flat file to backupdir for Union TN from Southern\n' -1
mv "$f" "$backupdir/$date_dir/$backup_file"
printf '%(%c)T - Copy backup file to the Union TN Output Directory\n' -1
cp "$backupdir/$date_dir/$backup_file" "$outputdir/$date_dir/$backup_file"
done
When using a glob with mv, the target must be an existing directory, and all matching files will be moved inside that directory.
In your case,
mv $INPUTDIR/* $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt
tells mv to move all file inside the $INPUTDIR/* directory to a directory named $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt.
I'm not sure what you're trying to do, but I hope this help.
Some more advice you could use:
Don't put the shebang (the first line beginning with "#") and the first three variable declarations inside single-quotes.
Some argue it is more portable and better to write /usr/bin/env bash instead of /bin/bash in the shebang
if [ CONDITION ] /then ACTION /fi statements can be simplified by writing [ CONDITION ] && ACTION
You reduce your likely hood of encountering unexpected behaviour when double-quoting your strings and variable (i.e. write "${year}/${month}/" instead of $year/$month.
No need to call mkdir a, followed by mkidr a/b, then mkdir a/b/c and so on, you can just call mkdir -p a/b/c. The p flag tells mkdir to create parent directories if they don't already exist.
It is unnecessary to validate the existence of a directory before calling mkdir since mkdir already validates that for you.
As pointed out by commenters, all-caps variables are conventions for special POSIX related variables. You should use another type of casing.
You could use date to do the formatting for you: date +%Y/%m/%d will print 2021/10/25
Strings without interpolation can have single-quotes.
(Optional, prevent undesired behaviors) Put set -e at the beginning of your scripts, after the shebang, to tell bash to halt if an error is encountered
And finally, use man <command_name> for built-in documentation!
i have UNI work where i have to create a shell script that monitors a directory with a few text files that can inform you (can be manual check to be informed) if they have been modified. honestly im not sure where to start and im bad at linux and cant find anything to help. i can only use standard tools in ubuntu. any help would be great. thankyou
update - this is what i have so far and i need a way to verify that the values printed are the same after altering a file (if they are not the same then print whats files have been changed)
also sorry first time using the site trying to learn..
#!/bin/sh
echo "press 1 to check - press 2 to exit"
while :
do
read INPUT_STR
case $INPUT_STR in
1)
echo "checking sums"
md5sum Target/bob
md5sum Target/bec
md5sum Target/john
md5sum Target/mary
md5sum Target/mike
;;
2)
break
;;
*)
echo "incorrect input"
esac
done
echo "thankyou for using IDS"
You mean something like that?
#!/bin/bash
WORKDIR=/home
TARGETS=(
"$WORKDIR/bob"
"$WORKDIR/bec"
"$WORKDIR/john"
"$WORKDIR/mary"
"$WORKDIR/mike"
)
for target in "${TARGETS[#]}"; do
md5file="${target##*/}.md5"
if [[ -e "$md5file" ]]; then
md5sum --quiet -c "$md5file"
else
echo "create md5sum referenz file $md5file"
md5sum "$target"/* > "$md5file"
fi
done
At first run it creates for every directory a reference file. On the second run the directory will be compare with reference file. Modifications will be shown. When you delete one reference file, it will be created again on the next run.
explanation
# loop over the array with the targets
for target in "${TARGETS[#]}"; do
# remove all directories except the last, append .md5
# and assign it to the variable md5file
md5file="${target##*/}.md5"
# if md5file exists, use it as reference and check if
# modifications where happend
if [[ -e "$md5file" ]]; then
# use the md5file as reference and only show
# modified files, ignore ok messages
md5sum --quiet -c "$md5file"
else
# when no md5file exists
echo "create md5sum referenz file $md5file"
# use target, append /* to it and run md5sum
# redirect output to md5file (reference file for next run)
md5sum "$target"/* > "$md5file"
fi
done
I have the following bash script, that I launch using the terminal.
dataset_dir='/home/super/datasets/Carpets_identification/data'
dest_dir='/home/super/datasets/Carpets_identification/augmented-data'
# if dest_dir does not exist -> create it
if [ ! -d ${dest_dir} ]; then
mkdir ${dest_dir}
fi
# for all folder of the dataset
for folder in ${dataset_dir}/*; do
curr_folder="${folder##*/}"
echo "Processing $curr_folder category"
# get all files
for item in ${folder}/*; do
# if the class dir in dest_dir does not exist -> create it
if [ ! -d ${dest_dir}/${curr_folder} ]; then
mkdir ${dest_dir}/${curr_folder}
fi
# for each file
if [ -f ${item} ]; then
# echo ${item}
filename=$(basename "$item")
extension="${filename##*.}"
filename=`readlink -e ${item}`
# get a certain number of patches
for i in {1..100}
do
python cropper.py ${filename} ${i} ${dest_dir}
done
fi
done
done
Given that it needs at least an hour to process all the files.
What happens if I change the '100' with '1000' in the last for loop and launch another instance of the same script?
Will the first process count to 1000 or will continue to count to 100?
I think the file will be readonly when a bash process executes it. But you can force the change. The already running process will count to its original value, 100.
You have to take care about the results. You are writing in the same output directory and have to expect side effects.
"When you make changes to your script, you make the changes on the disk(hard disk- the permanent storage); when you execute the script, the script is loaded to your memory(RAM).
(see https://askubuntu.com/questions/484111/can-i-modify-a-bash-script-sh-file-while-it-is-running )
BUT "You'll notice that the file is being read in at 8KB increments, so Bash and other shells will likely not load a file in its entirety, rather they read them in in blocks."
(see https://unix.stackexchange.com/questions/121013/how-does-linux-deal-with-shell-scripts )
So, in your case, all your script is loaded in the RAM memory by the script interpretor, and then executed. Meaning that if you change the value, then execute it again, the first instance will still have the "old" value.
I'm trying to extract a RAR file in a destination directory with the unrar command, but I don't want to display the normal file list that unrar shows when executing. What I want to do instead, is show a progress bar that shows the process in percentages using a dialog --gauge, but so far I only get the error
syntax error: unexpected end of file
Here's the code I'm using:
#!/bin/bash
dest="testing"
file"example.rar"
cmd="$(unrar x "$file" "$dest")" &>/dev/null
fc="$(unrar l "$file" | wc -l)" # get file count inside RAR file
fc="$((fc-10))" # file count has 10 extra lines, remove them
i=0 # index counter
(
for f in "$cmd"; do
pct="$(( 100*(++i)/fc ))" # calculate percentage
cat <<EOF
XXX
$pct
Extracting file "$f"...
XXX
EOF
done
) | dialog --title "Extracting Files" --gauge "Please wait" 7 70 0
So far, the code extracts the files in the dest folder, but they don't show the dialog box, and show the error unexpected end of file after completing the file extraction. It doesn't show any files being extracted which is to be expected by using &>/dev/null but I cannot get the dialog box to display properly.
I got this code from various examples I found online, but so far I cannot make it work. Can someone point me in the right direction?
Well, since no one has answered my question, I'm gonna answer it myself. I Finally managed to make the program work, it extracts the files, prints the pathnames of the files being extracted, and display the gauge bar filling correctly as each file is extracted. Here is the code:
#!/bin/bash
dest="testing" # destination directory
file"example.rar"
(
fc="$(unrar l "$file" | wc -l)" # get file count inside RAR file
fc="$((fc-10))" # file count has 10 extra lines, remove them
i=0 # index counter
unrar x "$file" "$dest" | while IFS="" read -r in ; do
# extract the pathname of file being extracted
f=$(echo $in | cut -d ' ' -f 2)
echo "XXX"
echo "$f..." # Display file name in the dialog box
echo "XXX"
echo "$pct %" # Show percent
pct="$(( 100*(++i)/fc ))" # Calculate current percent
done
) | dialog --title "Extracting Files" --gauge "Please wait" 8 75 0
This next part is supposed to capture the result of executing the extraction of the .RAR file, however I'm unsure if at this point I'm just capturing the result of the execution of the dialog box or that of the unrar command. What I want to do is control the flow the script in case the unrar command fails, I'm not entirely sure of how I should proceed, but right now, after the line where I display the dialog, I put the following if-then:
if [ "$?" ]; then
clear
echo "Command successful"
fi
Is this correct or should I move the if-then block to somewhere else in the code?
In the meantime, Imma work in a way to make this script work like a kind of prettied up unrar command, where you can specify the file name to be extracted and an optional destination directory, that way I'll be able to eliminate the need to specify a destination directory
Edit:
This is the current code I'm using for the solution of my own question. So far the code seems to be handling errors just fine. I have tested it only with the "File already exist" kind of errors that halts the execution of the script to wait for input from the user, and CRC errors for the cases where you have partitioned the compressed files in various parts and one of them is missing.
#!/bin/bash
dest="testing" # destination directory
file="example.rar"
(
fc="$(unrar l "$file" | wc -l)" # get file count inside RAR file
fc="$((fc-10))" # file count has 10 extra lines, remove them
i=0 # index counter
# I have to redirect the output from 0 to 1 because when there's a
# "file already exists" kind of error, unrar halts the execution of the
# script to wait for user input, so I have to process that message inside
# of the loop
unrar x "$file" "$dest" 0>&1 2>&1 | while IFS="" read -r in ; do
# we got a "file already exists" message?
if [[ "$f" == *"already"* ]]; then
echo "Error" # display an error message
break # exit the while loop
fi
# extract the pathname of file being extracted
f=$(echo $in | cut -d ' ' -f 2)
echo "XXX"
echo "$f..." # Display file name in the dialog box
echo "XXX"
echo "$pct %" # Show percent
pct="$(( 100*(++i)/fc ))" # Calculate current percent
done
exit ${PIPESTATUS[0]} # exit with status message we got from unrar
) | dialog --title "Extracting Files" --gauge "Please wait" 8 75 0
# if PIPESTATUS[0] is not equal to zero, it means there was an error
if [[ "${PIPESTATUS[0]}" -ne 0 ]]; then
echo "There was an error." # display a message
exit 1 # exit with status 1
fi
# if command was successful, display a message
if (( "$?" == 0 )); then
echo "Command executed correctly $?"
fi
So far, I'm not doing anything with the errors, I'm just interested in detecting them and returning messages to the scripts that call this one. It seems to be working just fine, but I know there has to be a better way to do this. What are your thoughts?
I have written a small bash (4) script to backup shares from my windows pc's. Currently I backup only one share and the backup is only visible to root. Please give me some hints for improvements of that piece of code:
#!/bin/bash
# Script to back up directories on several windows machines
# Permissions, owners, etc. are preserved (-av option)
# Only differences are submitted
# Execute this script from where you want
# Make sure only root can run our script
if [ "$(id -u)" != "0" ]; then
echo "This script must be run as root" 1>&2
exit 1
fi
# Specify the current date for the log-file name
current_date=$(date +%Y-%m-%d_%H%M%S)
# Specify the path to a list of file patterns not included in backup
script_path=$(dirname $(readlink -f $0))
rsync_excludes=$script_path/rsync_exclude.patterns
# Specify mount/rsync options
credential_file="/root/.smbcredentials"
# Specify windows shares
smb_shares=( //192.168.0.100/myFiles )
# Specify the last path component of the directory name to backup shares
# content into
smb_share_ids=( "blacksmith" )
# Specify with trailing '/' to transfer only the dir content
rsync_src="/mnt/smb_backup_mount_point/"
rsync_dst_root=(~/BACKUPS)
# Check if all arrays have the same size
if [ "${#smb_shares[#]}" -ne "${#smb_share_ids[#]}" ]; then
echo "Please specify one id for each samba share!"
exit 1
fi
# Run foor loop to sync all specified shares
for (( i = 0 ; i < ${#smb_shares[#]} ; i++ ))
do
# Check if mount point already exists
echo -n "Checking if mount point exists ... "
if [ -d $rsync_src ]; then
echo "Exists, exit!"
exit 1
else
echo "No, create it"
mkdir $rsync_src
fi
# Try to mount share and perform rsync in case of success
echo -n "Try to mount ${smb_shares[$i]} to $rsync_src ... "
mount -t cifs ${smb_shares[$i]} $rsync_src -o credentials=/root/.smbcredentials,iocharset=utf8,uid=0,file_mode=0600,dir_mode=0600
if [ "$?" -eq "0" ]; then
echo "Success"
# Specify the log-file name
rsync_logfile="$rsync_dst_root/BUP_${smb_share_ids[$i]}_$current_date.log"
# Build rsync destination root
rsync_dst=( $rsync_dst_root"/"${smb_share_ids[$i]} )
# Check if rsync destination root already exists
echo -n "Check if rsync destination root already exists ... "
if [ -d $rsync_dst ]; then
echo "Exists"
else
echo "Does not exist, create it"
mkdir -p $rsync_dst
fi
# Run rsync process
# -av > archieve (preserve owner, permissions, etc.) and verbosity
# --stats > print a set of statistics showing the effectiveness of the rsync algorithm for your files
# --bwlimit=KBPS > transfer rate limit - 0 defines no limit
# --progress > show progress
# --delete > delete files in $DEST that have been deletet in $SOURCE
# --delete-after > delete files at the end of the file transfer on the receiving machine
# --delete-excluded > delete excluded files in $DEST
# --modify-window > files differ first after this modification time
# --log-file > save log file
# --exclude-from > exclude everything from within an exclude file
rsync -av --stats --bwlimit=0 --progress --delete --delete-after --delete-excluded --modify-window=2 --log-file=$rsync_logfile --exclude-from=$rsync_excludes $rsync_src $rsync_dst
fi
# Unmount samba share
echo -n "Unmount $rsync_src ... "
umount $rsync_src
[ "$?" -eq "0" ] && echo "Success"
# Delete mount point
echo -n "Delete $rsync_src ... "
rmdir $rsync_src
[ "$?" -eq "0" ] && echo "Success"
done
Now I need some help concerning following topics:
Checking if conditions like share existence, mount point existence (to make a fool proof script)
The mount command - is it correct, do I give the correct permissions?
Is there a better place for the backup files than the home directory of a user if only root can see that files?
Do you think it would be helpful to integrate the backup of other file systems too?
The backup is rather slow (around 13mb/s) although I have a gigabit network system - possibly this is because of the ssh encryption of rsync? The linux system, where the share is mounted on, has a pci sata controller and an old mainboard, 1gb ram and an athlon xp 2400+. Could there be other reasons for the slowness?
If you have more topics that can be addressed here - be welcome to post them. I'm interested =)
Cheers
-Blackjacx
1) There is a lot of error checking you can do to make this script more robust: check for existance and execute status of external executables (rsync, id, date, dirname, etc.), check the output of rsync (with if [ 0 -ne $?]; then), ping the remote machine to make sure it is on the network, check to make sure thet user you are doing the backup for has a home directory on the local machine, check to make sure the destination directory has enough space for the rsync, etc.
2) mount command looks ok, did you want to try to mount as read only?
3) Depends on the number of users and the size of the backup. For small backups the home directory is likely ok, but if the backups are large, especialy if you might run out of disk space, a dedicated backup location would be nice.
4) Depends on what the purpose of the backup is. In my experience there are five kinds of things that people backup: user data, system data, user configurations, system configurations, entire disks or filesystems. Is there a seperate system task that backs-up the system data and configurations?
5) What other tasks are running while the backup takes place? If the backup is running automated at say 1 AM, there may be other system intensive tasks scheduled for the same time. What is your typical throughput rate for other kinds of data?
You are performing an array check on non-array variables.
Use a variable to specify the remote machine.
# Remote machine
declare remote_machine="192.168.0.100"
# Specify windows shares array
declare -a smb_shares=([0]="//$remote_machine/myFiles"
[1]="//$remote_machine/otherFiles" )
# Specify the last path component of the directory name to backup shares
# content into
declare -a smb_share_ids=( [1]="blacksmith" [2]="tanner")
that way you can
ping -c 1 $remote_machine
if [ 0 -ne $? ] ; then
echo "ping on $remote_machine Failed, exiting..."
exit 1
fi
If rsync_src is only used for this mackup, you might wat to try
mnt_dir="/mnt"
rsync_src=`mktemp -d -p $mnt_dir`
and I would make this dir before the for loop otherwise you are creating and destroying it for every directory you back up.