I have a script which calls the rsync command with some dynamically generated parameters but I'm having trouble passing them correctly.
Here's some excerpt:
logfile="$logDir/$(timestamp) $name.log"
echo "something" >> "$logfile"
params="-aAXz --stats -h --delete --exclude-from $exclude --log-file=$logfile $src $dest"
if [ "$silent" = "" ]; then
params="-v $params --info=progress2"
fi
rsync $params
If the logfile is e.g. /tmp/150507 test.log, the something statement is actually written to /tmp/150507 test.log, but rsync writes its logs to /tmp/150507 (everything after the first blank removed).
If I explicitly quote the name of the logfile in the params, rsync throws an exception:
params="-aAXz --stats -h --delete --exclude-from $exclude --log-file=\"$logfile\" $src $dest"
The error:
rsync: failed to open log-file "/tmp/150507: No such file or directory (2)
Ignoring "log file" setting.
How can I generate the params dynamically without losing the ability to use blanks in the filenames?
More quoting needed around log file name:
declare -a params
params=(-aAXz --stats -h --delete --exclude-from "$exclude" --log-file="$logfile" "$src" "$dest")
if [[ "$silent" = "" ]]; then
params=( -v "${params[#]}" --info=progress2 )
fi
rsync "${params[#]}"
This is the case where you should consider using BASH arrays to constitute a dynamic command line.
Related
I have written a bash script to backup my project directory but the exclude option isn't working.
backup.sh
#!/bin/sh
DRY_RUN=""
if [ $1="-n" ]; then
DRY_RUN="n"
fi
OPTIONS="-a"$DRY_RUN"v --delete --delete-excluded --exclude='/bin/'"
SOURCE="/home/vikram/Documents/sem4/oop/lab/java_assignments/student_information_system/"
DEST="/home/vikram/Documents/sem4/oop/lab/java_assignments/student_information_system_backup"
rsync $OPTIONS $SOURCE $DEST
When I am executing the command separately on the terminal, it works.
vikram:student_information_system$ rsync -anv --delete --delete-excluded --exclude='/bin/' /home/vikram/Documents/sem4/oop/lab/java_assignments/student_information_system/ /home/vikram/Documents/sem4/oop/lab/java_assignments/student_information_system_backup
sending incremental file list
deleting bin/student_information_system/model/StudentTest.class
deleting bin/student_information_system/model/Student.class
deleting bin/student_information_system/model/
deleting bin/student_information_system/
deleting bin/
./
.backup.sh.swp
backup.sh
backup.sh~
sent 507 bytes received 228 bytes 1,470.00 bytes/sec
total size is 16,033 speedup is 21.81 (DRY RUN)
vikram:student_information_system$
The single quotes around the name of the directory which is to be excluded were causing the problem(explained in this answer).
Also I stored all the options in an array as explained here.
Removing the single quotes, storing the options in an array, double quoting the variables as suggested by #Cyrus in the comments, solved the problem.
Also I had to change #!/bin/sh to #!/bin/bash.
Updated script:
#!/bin/bash
DRY_RUN=""
if [ "$1" = "-n" ]; then
DRY_RUN="n"
fi
OPTS=( "-a""$DRY_RUN""v" "--delete" "--delete-excluded" "--exclude=/bin/" )
SRC="/home/vikram/Documents/sem4/oop/lab/java_assignments/student_information_system/"
DEST="/home/vikram/Documents/sem4/oop/lab/java_assignments/student_information_system_backup"
echo "rsync ${OPTS[#]} $SRC $DEST"
rsync "${OPTS[#]}" "$SRC" "$DEST"
I've written a bash script to sync backups with a local storage, the script checks whether a backup as been made on the day the script is executed, and if so it sync's.
I've made it this way so that if by accident (or other way) all the backups are deleted from the original location, the synced backups on the second storage won't be deleted upon next sync.
#!/bin/bash
files_found=`ssh user#xx.xx.xx.xx "find /home/data_folder/test* -type f -mtime -1"`
rsync_to_location="/home/test_folder/";
rsync_from_location="/home/data_folder/";
if [ "$files_found" = 0 ]; then
echo "File not found!"
send_error=`ssh user#xx.xx.xx.xx "echo 'This is the message body' | mail -s 'This is the subject' user#localhost"`
else
echo "Files found!"
rsync -arzt --ignore-existing --delete --max-delete=1 -e 'ssh' user#xx.xx.xx.xx:$rsync_from_location $rsync_to_location
if [[ $? -gt 0 ]]; then
send_error=`ssh user#xx.xx.xx.xx "echo 'This is the message body' | mail -s 'This is the subject' earmaster#localhost"`
fi
fi
Now my question is, if the rsync fails (max deletions), how can I store that message and send it with the mail?
I've tried with
rsync_error="rsync -arzt --ignore-existing --delete --max-delete=1 -e 'ssh' user#xx.xx.xx.xx:$rsync_from_location $rsync_to_location"
And then add the $rsync_error to the mail call, but it doesn't seem to work
The line you have put here will just store that command as a string and not run it.
rsync_error="rsync -arzt --ignore-existing --delete --max-delete=1 -e 'ssh' user#xx.xx.xx.xx:$rsync_from_location $rsync_to_location"
To capture its output you would need to put it in a $( ) like so.
rsync_error=$(rsync -arzt --ignore-existing --delete --max-delete=1 -e 'ssh' user#xx.xx.xx.xx:$rsync_from_location $rsync_to_location)
This will capture the stdout of the executed command but you are wanting stderr I assume. So a better way of doing this might be to pipe to stderr to a file and handle the output that way.
# rsync err output file
ERR_OUTPUT="/tmp/rsync_err_$$"
# When the script exits, remove the file
trap "rm -f $ERR_OUTPUT" EXIT
# Use 2> to direct stderr to the output file
rsync -arzt --ignore-existing --delete --max-delete=1 -e 'ssh' user#xx.xx.xx.xx:$rsync_from_location $rsync_to_location 2> "$ERR_OUTPUT"
# Do what ever with your error file
I have been working on a backup script that uses rsync to do an incremental backup.
I have tested the following rsync command manually, and it runs and completes a backup without error:
rsync -aAXv --delete --progress --link-dest=/backup/Uyuk/Uyuk-backup-part1/2014-02-24/ /mnt/backup/ /backup/Uyuk/Uyuk-backup-part1/2014-02-25/
however when I run that same command in my backup script it gives me the following error:
rsync: -aAXv --delete --progress --link-dest=/backup/Uyuk/Uyuk-backup-part1/2014-02-24/ /mnt/backup/ /backup/Uyuk/Uyuk-backup-part1/2014-02-25/: unknown option
rsync error: syntax or usage error (code 1) at main.c(1422) [client=3.0.6]
I ran bash -x on my script to figure out exactly what is sent to the console and here is what was printed:
+ rsync '-aAXv --delete --progress --link-dest=/backup/Uyuk/Uyuk-backup-part1/2014-02-24/ /mnt/backup/ /backup/Uyuk/Uyuk-backup-part1/2014-02-25/'
Does anyone see what is wrong? I cant find anything that would cause the syntax error.
EDIT:
Here is the actual code I have in the script, and this is a pretty large script so yes some variables are not defined here, but you get the idea.
mkdir -p "/backup/$HOST/$NAME/$TODAY"
#source directory
SRC="$MNT"
#link directory
LNK="/backup/$HOST/$NAME/$LAST/"
#target directory
TRG="/backup/$HOST/$NAME/$TODAY/"
#rsync options
OPT1="-aAXv --delete --progress --link-dest=$LNK"
#run the rsync command
echo "rsync $OPT1 $SRC $TRG"
rsync "$OPT1 $SRC $TRG" > /var/log/backup/backup.rsync.log 2>&1
You are passing your option list as a single argument, when it needs to be passed as a list of arguments. In general, you should use an array in bash to hold your arguments, in case any of them contain whitespace. Try the following:
mkdir -p "/backup/$HOST/$NAME/$TODAY"
#source directory
SRC="$MNT"
#link directory
LNK="/backup/$HOST/$NAME/$LAST/"
#target directory
TRG="/backup/$HOST/$NAME/$TODAY/"
#rsync options
OPTS=( "-aAXv" "--delete" "--progress" "--link-dest=$LNK" )
#run the rsync command
echo "rsync $OPT1 $SRC $TRG"
rsync "${OPTS[#]}" "$SRC" "$TRG" > /var/log/backup/backup.rsync.log 2>&1
An array expansion ${OPTS[#]}, when quoted, is treated specially as a sequence of arguments, each of which is quoted individually to preserve any whitespace or special characters in the individual elements. If arr=("a b" c d), then echo "${arr[#]}" is the same as
echo "a b" "c" "d"
rather than
echo "a b c d"
This will not work in a shell that doesn't support arrays, but then, arrays were invented because there wasn't a safe way (that is, without using eval) to handle this use case without them.
This:
rsync "$OPT1 $SRC $TRG"
passes all your intended arguments lumped together as one argument, which rsync doesn't know how to deal with.
Try this instead:
rsync ${OPT1} ${SRC} ${TRG}
The approach suggested by #chepner didn't work on my Mac OS X (10.9.4), but eval did.
eval rsync "$OPT1 $SRC $TRG"
I have this bash script:
#!/bin/bash
inotifywait -m -e close_write --exclude '\*.sw??$' . |
#adding --format %f does not work for some reason
while read dir ev file; do
cp ./"$file" zinot/"$file"
done
~
Now, how would I have it do the same thing but also handle deletes by writing the filenames to a log file?
Something like?
#!/bin/bash
inotifywait -m -e close_write --exclude '\*.sw??$' . |
#adding --format %f does not work for some reason
while read dir ev file; do
# if DELETE, append $file to /inotify.log
# else
cp ./"$file" zinot/"$file"
done
~
EDIT:
By looking at the messages generated, I found that inotifywait generates CLOSE_WRITE,CLOSE whenever a file is closed. So that is what I'm now checking in my code.
I tried also checking for DELETE, but for some reason that section of the code is not working. Check it out:
#!/bin/bash
fromdir=/path/to/directory/
inotifywait -m -e close_write,delete --exclude '\*.sw??$' "$fromdir" |
while read dir ev file; do
if [ "$ev" == 'CLOSE_WRITE,CLOSE' ]
then
# copy entire file to /root/zinot/ - WORKS!
cp "$fromdir""$file" /root/zinot/"$file"
elif [ "$ev" == 'DELETE' ]
then
# trying this without echo does not work, but with echo it does!
echo "$file" >> /root/zinot.txt
else
# never saw this error message pop up, which makes sense.
echo Could not perform action on "$ev"
fi
done
In the dir, I do touch zzzhey.txt. File is copied. I do vim zzzhey.txt and file changes are copied. I do rm zzzhey.txt and the filename is added to my log file zinot.txt. Awesome!
You need to add -e delete to your monitor, otherwise DELETE events won't be passed to the loop. Then add a conditional to the loop that handles the events. Something like this should do:
#!/bin/bash
inotifywait -m -e delete -e close_write --exclude '\*.sw??$' . |
while read dir ev file; do
if [ "$ev" = "DELETE" ]; then
echo "$file" >> /inotify.log
else
cp ./"$file" zinot/"$file"
fi
done
For some reason this code creates problems:
source="/foo/bar/"
destination="/home/oni/"
if [ -d $source ]; then
echo "Source directory exists"
if [ -d $destination ]; then
echo "Destination directory exists"
rsync -raz --delete --ignore-existing --ignore-times --size-only --stats --progress $source $destination
chmod -R 0755 $destination
else
echo "Destination directory does not exists"
fi
else
echo "Source directory does not exists"
fi
It errors out with:
Source directory exists
/usr/bin/copyfoo: line 7: [: too many arguments
Destination directory does not exists
I used nested if statements in bash before without a problem, what simple mistake am I overlooking?
Thanks!
The syntax really looks correct. Works in dash/bash.
Did you change the name of the destination directory for this example? If your real name contains e.g. whitespace you're better off quoting the tested variable.
if [ -d "$destination" ]; then
(I would leave the destination-directory-check anyway as rsync will create the directory if its missing. If you copy on the same computer and not over a network, I would leave the -z compression parameter of rsync also.)
Update:
Does this work for you? (You'll have to change paths)
#!/bin/bash
source="/tmp/bar/"
destination="/tmp/baz/"
test -d "$source" || {
echo "$source does not exist"
exit
}
rsync -ra \
--delete \
--ignore-existing --ignore-times --size-only \
--stats --progress "$source" "$destination"
if [ "$?" -gt 0 ]; then
echo "Failure exit value: $?"
fi
I suspect your destination is set to something different than you showed us above, probably something containing whitespace.
You should put double quotes around it in the [ ] block too, e.g. [ -d "$destination" ].