I'm writing a bash script to collect some directories (where conditions are met) and rsync those to a remote location. Overall the script looks like this;
sources=""
for d in /somewhere/* ; do
if $d meets condition; then
sources="$sources $(printf %q "$d")"
fi
done
if [ ! -z $sources ] ; then
rsync -vrz $sources /remote/target/
fi
Note that I'm using printf %q to escape spaces in directory names.
However when there are spaces in directory names, for example when "/somewhere/dir name" met the condition, the rsync thinks that as two directories and fails to run;
(at /home/u/) $ bash script.sh
sending incremental file list
rsync: link_stat "/somewhere/dir\" failed: No such file or directory (2)
rsync: link_stat "/home/u/name" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]
If I just print the rsync command by changing the last line to
echo rsync -vrz $sources /remote/target/
it looks just fine.
(at /home/u/) $ bash script.sh
rsync -vrz /somewhere/dirname /somewhere/dir\ name /remote/target
But using set -x shows something wacky going on.
(at /home/u/) $ bash script.sh
+ rsync -vrz /somewhere/dirname '/somewhere/dir\' name /remote/target
sending incremental file list
rsync: link_stat "/somewhere/dir\" failed: No such file or directory (2)
rsync: link_stat "/home/u/name" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]
I also tried to use double quoted directory names instead of printf %q but it didn't work either, with a slightly different reason.
(at /home/u/) $ bash script.sh
+ rsync -vrz '"/somewhere/dirname"' '"/somewhere/dir' 'name"' /remote/target
sending incremental file list
rsync: change_dir "/home/u//"/somewhere" failed: No such file or directory (2)
rsync: change_dir "/home/u//"/somewhere" failed: No such file or directory (2)
rsync: link_stat "/home/u/name"" failed: No such file or directory (2)
sent 20 bytes received 12 bytes 64.00 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]
Where are those single quotes around some arguments coming from and what is the best way to collect directories with spaces in a single-lined variable for using as sources in cp, mv, or rsync?
Use an array instead.
sources=()
for d in /somewhere/* ; do
if $d meets condition; then
sources+=("$d")
fi
done
if [ "${sources[*]}" ]; then
rsync -vrz "${sources[#]}" /remote/target/
fi
The single quotes output with set -x are just for disambiguation, so you can see exactly where actual literal spaces go (as opposed to syntactic spaces which are argument separators).
Related
This a really short question. But is there something syntatically wrong with placing a variable $example as an argument for tar in a bash file?
I have the file written as
//only portion that really matters
#!/bin/bash
...
tar -cvpzf $filename $backup_source
//here's the actual code
#!/bin/bash
backup_source="~/momobobo"
backup_dest="~/momobobo_backup/"
dater=`date '+%m-%d-%Y-%H-%M-%S'`
filename="$backup_dest$dater.tgz"
echo “Backing Up your Linux System”
tar -cvpzf $filename $backup_source
echo tar -cvpzf $filename $backup_source
echo “Backup finished”
//and heres the error
“Backing Up your Linux System”
tar: ~/momobobo: Cannot stat: No such file or directory
tar (child): ~/momobobo_backup/07-02-2013-18-34-12.tgz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
tar -cvpzf ~/momobobo_backup/07-02-2013-18-34-12.tgz ~/momobobo
Notice the "echo tar ...". When I copy and paste the output and run it in my terminal there is no problem taring the file. I'm currently running Xubuntu and I already did an update.
~ doesn't expand to your home directory in double quotes.
Just remove the double quotes:
backup_source=~/momobobo
backup_dest=~/momobobo_backup/
In cases where you have things you would want to quote, you can use ~/"momobobo"
This question already has answers here:
Why does shell ignore quoting characters in arguments passed to it through variables? [duplicate]
(3 answers)
Why do bash parameter expansions cause an rsync command to operate differently?
(2 answers)
Rsync syntax error when run from bash script
(3 answers)
Closed 3 years ago.
All the other solutions on SO work well for echo and similar commands. However, I am trying to assign a long list of arguments (split across multiple lines) to a single variable, which will be expanded into multiple arguments for an rsync command. Something about the rsync command (or bash's whitespace tokenizing) doesn't work well with most of the recommended solutions.
SYNC_OPTIONS="-a --progress --delete --delete-before --delete-excluded"
SYNC_EXCLUDE="
--exclude='some/dir/with spaces/requiring quoting'
--exclude='another/dir/with spaces'
--exclude='blah blah/blah'
"
echo rsync ${SYNC_OPTIONS} ${SYNC_EXCLUDE} /src/ /dst/
set -x
rsync ${SYNC_OPTIONS} ${SYNC_EXCLUDE} /src/ /dst/
set +x
Which produces:
$ echo rsync ${SYNC_OPTIONS} ${SYNC_EXCLUDE} /src/ /dst/
rsync -a --progress --delete --delete-before --delete-excluded --exclude='some/dir/with spaces/requiring quoting' --exclude='another/dir/with spaces' --exclude='blah blah/blah' /src/ /dst/
$ set -x
$ rsync ${SYNC_OPTIONS} ${SYNC_EXCLUDE} /src/ /dst/
+ rsync -a --progress --delete --delete-before --delete-excluded '--exclude='\''some/dir/with' spaces/requiring 'quoting'\''' '--exclude='\''another/dir/with' 'spaces'\''' '--exclude='\''blah' 'blah/blah'\''' /src/ /dst/
rsync: change_dir "/spaces" failed: No such file or directory (2)
rsync: link_stat "/quoting'" failed: No such file or directory (2)
rsync: link_stat "/spaces'" failed: No such file or directory (2)
rsync: change_dir "/blah" failed: No such file or directory (2)
Both the tracing output and the rsync error messages indicate that there is some extra, unwanted level of tokenizing, which is breaking the usage of the variable in the rsync command. However, the echo command's output looks correct.
How can I define a multi-line variable, which will be used as an argument for the rsync command, where the argument contains quoted file paths with spaces? Specifically, I'm interested in the above case for the SYNC_EXCLUDE variable.
Finally found answer embedded in lengthy, but good answer here. For my case, I converted the variable to array of strings, instead of a single long string, like so:
SYNC_OPTIONS="-a --progress --delete --delete-before --delete-excluded"
SYNC_EXCLUDE=(
--exclude='some/dir/with spaces/requiring quoting'
--exclude='another/dir/with spaces'
--exclude='blah blah/blah'
)
echo rsync ${SYNC_OPTIONS} "${SYNC_EXCLUDE[#]}" /src/ /dst/
set -x
rsync ${SYNC_OPTIONS} "${SYNC_EXCLUDE[#]}" /src/ /dst/
set +x
I am trying to perform a set of terminal operations using bash shell script. Below is my code
#!/bin/bash
FILE_DATE=`date '+%Y%m%d'`
ARCHIVE_DIR="/home/tanmay/backup/"
TAR_GZ=".tar.gz"
PATH=( "/home/tanmay/Downloads/apache-tomcat-7.0.69/logs" "/home/tanmay/Downloads/apache-tomcat-7.0.69/webapps" )
FOLDER=("logs" "webapps" )
for number in {0..1..1}
do
echo ${PATH[number]}
echo ${FOLDER[number]}
rsync -vrzh ${PATH[number]} ${ARCHIVE_DIR}
tar -zcvf ${ARCHIVE_DIR}/${FOLDER[number]}${TAR_GZ} ${ARCHIVE_DIR}/${FOLDER[number]}
rm -rf ${ARCHIVE_DIR}/${FOLDER[number]}
if [ -f ${ARCHIVE_DIR}/${FOLDER[number]}${TAR_GZ} ]
then
mv ${ARCHIVE_DIR}${FOLDER[number]}${TAR_GZ} ${ARCHIVE_DIR}${FOLDER[number]}_${FILE_DATE}${TAR_GZ}
fi
done
When I run this script, both the echo is showing the correct values. But operations (rsync,tar..) are returning file not found. Below is the output
/home/tanmay/Downloads/apache-tomcat-7.0.69/logs
logs
./server_data_backup_updated.sh: line 11: rsync: No such file or directory
./server_data_backup_updated.sh: line 12: tar: No such file or directory
./server_data_backup_updated.sh: line 13: rm: No such file or directory
/home/tanmay/Downloads/apache-tomcat-7.0.69/webapps
webapps
./server_data_backup_updated.sh: line 11: rsync: No such file or directory
./server_data_backup_updated.sh: line 12: tar: No such file or directory
./server_data_backup_updated.sh: line 13: rm: No such file or directory
UPDATE 1
Using one array instead on two. It is working now.
#!/bin/bash
FILE_DATE=`date '+%Y%m%d'`
ARCHIVE_DIR="/home/tanmay/backup/"
TAR_GZ=".tar.gz"
array=( "/home/tanmay/Downloads/apache-tomcat-7.0.69/logs"
"logs"
"/home/tanmay/Downloads/apache-tomcat-7.0.69/webapps"
"webapps")
for number in {0..2..2}
do
rsync -vrzh ${array[number]} ${ARCHIVE_DIR}
tar -zcvf ${ARCHIVE_DIR}/${array[number+1]}${TAR_GZ} ${ARCHIVE_DIR}/${array[number+1]}
rm -rf ${ARCHIVE_DIR}/${array[number+1]}
if [ -f ${ARCHIVE_DIR}/${array[number+1]}${TAR_GZ} ]
then
mv ${ARCHIVE_DIR}${array[number+1]}${TAR_GZ} ${ARCHIVE_DIR}${array[number+1]}_${FILE_DATE}${TAR_GZ}
fi
done
When you reset PATH, the shell can no longer find the executable rsync. When the shell reads the word rsync, it looks through the variable named PATH (which it expects to be a colon separated list of directories, not an array) for a file named rsync. Similarly for tar and rm. The error messages you see are simply telling you that those commands are not found in your PATH.
I have been working on a backup script that uses rsync to do an incremental backup.
I have tested the following rsync command manually, and it runs and completes a backup without error:
rsync -aAXv --delete --progress --link-dest=/backup/Uyuk/Uyuk-backup-part1/2014-02-24/ /mnt/backup/ /backup/Uyuk/Uyuk-backup-part1/2014-02-25/
however when I run that same command in my backup script it gives me the following error:
rsync: -aAXv --delete --progress --link-dest=/backup/Uyuk/Uyuk-backup-part1/2014-02-24/ /mnt/backup/ /backup/Uyuk/Uyuk-backup-part1/2014-02-25/: unknown option
rsync error: syntax or usage error (code 1) at main.c(1422) [client=3.0.6]
I ran bash -x on my script to figure out exactly what is sent to the console and here is what was printed:
+ rsync '-aAXv --delete --progress --link-dest=/backup/Uyuk/Uyuk-backup-part1/2014-02-24/ /mnt/backup/ /backup/Uyuk/Uyuk-backup-part1/2014-02-25/'
Does anyone see what is wrong? I cant find anything that would cause the syntax error.
EDIT:
Here is the actual code I have in the script, and this is a pretty large script so yes some variables are not defined here, but you get the idea.
mkdir -p "/backup/$HOST/$NAME/$TODAY"
#source directory
SRC="$MNT"
#link directory
LNK="/backup/$HOST/$NAME/$LAST/"
#target directory
TRG="/backup/$HOST/$NAME/$TODAY/"
#rsync options
OPT1="-aAXv --delete --progress --link-dest=$LNK"
#run the rsync command
echo "rsync $OPT1 $SRC $TRG"
rsync "$OPT1 $SRC $TRG" > /var/log/backup/backup.rsync.log 2>&1
You are passing your option list as a single argument, when it needs to be passed as a list of arguments. In general, you should use an array in bash to hold your arguments, in case any of them contain whitespace. Try the following:
mkdir -p "/backup/$HOST/$NAME/$TODAY"
#source directory
SRC="$MNT"
#link directory
LNK="/backup/$HOST/$NAME/$LAST/"
#target directory
TRG="/backup/$HOST/$NAME/$TODAY/"
#rsync options
OPTS=( "-aAXv" "--delete" "--progress" "--link-dest=$LNK" )
#run the rsync command
echo "rsync $OPT1 $SRC $TRG"
rsync "${OPTS[#]}" "$SRC" "$TRG" > /var/log/backup/backup.rsync.log 2>&1
An array expansion ${OPTS[#]}, when quoted, is treated specially as a sequence of arguments, each of which is quoted individually to preserve any whitespace or special characters in the individual elements. If arr=("a b" c d), then echo "${arr[#]}" is the same as
echo "a b" "c" "d"
rather than
echo "a b c d"
This will not work in a shell that doesn't support arrays, but then, arrays were invented because there wasn't a safe way (that is, without using eval) to handle this use case without them.
This:
rsync "$OPT1 $SRC $TRG"
passes all your intended arguments lumped together as one argument, which rsync doesn't know how to deal with.
Try this instead:
rsync ${OPT1} ${SRC} ${TRG}
The approach suggested by #chepner didn't work on my Mac OS X (10.9.4), but eval did.
eval rsync "$OPT1 $SRC $TRG"
I am trying to save some typing with this bash script:
#!/usr/bin/env bash
n=$(($#-1))
files=${#:1:$n}
dest=${!#}
echo "$n files"
echo "${files[#]}"
echo "$dest"
rsync -av "${files[#]}" kyopti:$dest
Notice the "${files[#]}" part, I did this because it's recommended on Greg's bash tutorial. This generates an error:
rsync: link_stat "/tmp/scplot.pdf ssknplot.pdf" failed: No such file or directory (2)
But if I remove the quotes, everything works fine. I am confused, isn't this supposed to be the best practice, to add the quotes?
files is not an array. It's just a single string consisting of a space-separated list of the positional arguments. You want
files=( "${#:1:$n}" )
Then your quoted use of files in the rsync command will work as expected.