Rsync command variable, bash script - bash

RSYNC="rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete "
# Files
$RSYNC deploy#ip:/var/www/path1 /var/www/path1
$RSYNC deploy#ip:/var/www/path2 /var/www/path2
I'd like to introduce this RSYNC Variable to be more compact, but it throws an error:
Unexpected remote arg: deploy#ip:/var/www/path1
If i use only rsync inside the doublequotes, it works fine. For the sake of readability, i'd keep them separate command invocations.

I agree that eval is dangerous. In addition to the array approach #Eugeniu Rosca suggested, you could also use a shell function:
my_rsync() {
rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete "$#"
}
my_rsync deploy#ip:/var/www/path1 /var/www/path1
my_rsync deploy#ip:/var/www/path2 /var/www/path2
BTW, you should read BashFAQ #50: I'm trying to put a command in a variable, but the complex cases always fail!.

If you want to store the command string into a variable and evaluate it later, you can use eval or an alternative safer technique:
#!/bin/bash
# Store the command into an array
RSYNC=(rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete)
# Run the command
"${RSYNC[#]}" deploy#ip:/var/www/path1 /var/www/path1
Why eval should be avoided

After you assign a string to a variable and then submit it to shell again it is tokenized differently. Consider the following script
VAR="/bin/ls 'Test1 Test2'"
$VAR
It will throw two errors:
/bin/ls: cannot access 'Test1
/bin/ls: cannot access Test2'
You guessed it right, the apostrophes are no longer affecting tokenization. Instead they are treated as normal characters when the commandline arguments are fed to /bin/ls
While eval is rightly considered evil, it is by far the easiest way to get a script running. It stitches all the arguments into one line and applies the tokenizing procedure again.
RSYNC="rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete "
# Files
eval $RSYNC deploy#ip:/var/www/path1 /var/www/path1
eval $RSYNC deploy#ip:/var/www/path2 /var/www/path2

Related

Optionally pass an argument in a bash script

I would like to use custom identity file when using rsync, but only if the file exists, otherwise I don't want to bother with custom ssh command for rsync. I am having problems with quotes. See examples.
Desired command if identity file exists
rsync -e "ssh -i '/tmp/id_rsa'" /tmp/dir/ u#h:/tmp/dir
Desired command if identity file does not exist
rsync /tmp/dir/ u#h:/tmp/dir
I wanted to create a variable that would contain -e "ssh -i '/tmp/id_rsa'" and use it as follows
rsync ${identityArg} /tmp/dir/ u#h:/tmp/dir
This variable would be either empty or contain desired ssh command.
An example way I fill the variable (I have tried many ways)
IDENTITY_FILE="/tmp/id_rsa"
if [ -f "${IDENTITY_FILE}" ]; then
identityArg="-e 'ssh -i \"${IDENTITY_FILE}\"'"
fi
The problem is that quotes are always wrong in the command and I end up with commands similar to these ones (set -x is set in the script and this is the output)
rsync -e '\ssh' -i '"/tmp/id_rsa"'\''' /tmp/dir/ u#h:/tmp/dir
There is something I do not get about quotation in bash. If you have any good resource about usage of single and double quotes in bash script I would like to read it.
You want to add two positional parameters: -e and ssh -i '/tmp/id_rsa', where /tmp/id_rsa is an expanded variable. You should use an array for this:
args=(/tmp/dir/ u#h:/tmp/dir)
idfile=/tmp/id_rsa
# Let [[ ... ]] do the quoting
if [[ -f $idfile ]]; then
# Prepend two parameters to args array
args=(-e "ssh -i '$idfile'" "${args[#]}")
fi
rsync "${args[#]}"
I'm not convinced the inner single quotes are necessary for ssh -i, but this expands to exactly the commands shown in the question.
Try like this
id=/tmp/id_rsa
[[ -e $id ]] && o1='-e' o2="ssh -i '$id'"
echo $o1 "$o2" /tmp/dir/ u#h:/tmp/dir
Trying to properly escape quotes is tricky. Better to try to leverage existing constructs. Few alternatives, depending on the situations
If the name of the identity file only contain simple characters (no spaces, wildcard, etc.) consider not wrapping it in quotes. In this case, you can
IDENTITY_FILE="/tmp/id_rsa"
if [ -f "${IDENTITY_FILE}" ]; then
identityArg="-e 'ssh -i ${IDENTITY_FILE}'"
fi
...
rsync $identityArg ...
Another option is to always pass in the command (ssh or 'ssh -I ...'). This will automatically take care for special characters in the identity file.
IDENTITY_FILE="/tmp/id_rsa"
if [ -f "${IDENTITY_FILE}" ]; then
identityArg="-i '${IDENTITY_FILE}'"
fi
rsync -e "ssh $identityArg" ...
Third alternative is to use array to create the arguments to rsync, and let the shell escape the characters as needed. This will allow any character in the identity file.
IDENTITY_FILE="/tmp/id_rsa"
if [ -f "${IDENTITY_FILE}" ]; then
identityArg=(-e "ssh -i '${IDENTITY_FILE}'")
fi
rsync "${identityArg[#]}" ...

rsync backup script fails when options are in variable

I want to use rsync for incremental backups. However it fails when I try it like this:
SRC="/"
TRG="/backup/"
LNK="/oldbackup/"
OPT="-a --exclude={/dev,/proc,/sys,/tmp,/run,/mnt,/media,/lost+found} --link-dest=$LNK"
rsync $OPT $SRC $TRG
But works like this:
SRC="/"
TRG="/backup/"
LNK="/oldbackup/"
rsync -a --exclude={/dev,/proc,/sys,/tmp,/run,/mnt,/media,/lost+found} --link-dest=$LNK $SRC $TRG
What did I do wrong?
Check the order of expansions in man bash: brace expansion happens first, variable expansion happens later. Therefore, braces in a variable are not expanded.
You can use an array to capture the values, use expansion already in the assignment:
opts=( -a --exclude={/dev,/proc,/sys,/tmp,/run,/mnt,/media,/lost+found} --link-dest="$LNK" )
rsync "${opts[#]}" "$SRC" "$TRG"

Rsync syntax error when run from bash script

I have been working on a backup script that uses rsync to do an incremental backup.
I have tested the following rsync command manually, and it runs and completes a backup without error:
rsync -aAXv --delete --progress --link-dest=/backup/Uyuk/Uyuk-backup-part1/2014-02-24/ /mnt/backup/ /backup/Uyuk/Uyuk-backup-part1/2014-02-25/
however when I run that same command in my backup script it gives me the following error:
rsync: -aAXv --delete --progress --link-dest=/backup/Uyuk/Uyuk-backup-part1/2014-02-24/ /mnt/backup/ /backup/Uyuk/Uyuk-backup-part1/2014-02-25/: unknown option
rsync error: syntax or usage error (code 1) at main.c(1422) [client=3.0.6]
I ran bash -x on my script to figure out exactly what is sent to the console and here is what was printed:
+ rsync '-aAXv --delete --progress --link-dest=/backup/Uyuk/Uyuk-backup-part1/2014-02-24/ /mnt/backup/ /backup/Uyuk/Uyuk-backup-part1/2014-02-25/'
Does anyone see what is wrong? I cant find anything that would cause the syntax error.
EDIT:
Here is the actual code I have in the script, and this is a pretty large script so yes some variables are not defined here, but you get the idea.
mkdir -p "/backup/$HOST/$NAME/$TODAY"
#source directory
SRC="$MNT"
#link directory
LNK="/backup/$HOST/$NAME/$LAST/"
#target directory
TRG="/backup/$HOST/$NAME/$TODAY/"
#rsync options
OPT1="-aAXv --delete --progress --link-dest=$LNK"
#run the rsync command
echo "rsync $OPT1 $SRC $TRG"
rsync "$OPT1 $SRC $TRG" > /var/log/backup/backup.rsync.log 2>&1
You are passing your option list as a single argument, when it needs to be passed as a list of arguments. In general, you should use an array in bash to hold your arguments, in case any of them contain whitespace. Try the following:
mkdir -p "/backup/$HOST/$NAME/$TODAY"
#source directory
SRC="$MNT"
#link directory
LNK="/backup/$HOST/$NAME/$LAST/"
#target directory
TRG="/backup/$HOST/$NAME/$TODAY/"
#rsync options
OPTS=( "-aAXv" "--delete" "--progress" "--link-dest=$LNK" )
#run the rsync command
echo "rsync $OPT1 $SRC $TRG"
rsync "${OPTS[#]}" "$SRC" "$TRG" > /var/log/backup/backup.rsync.log 2>&1
An array expansion ${OPTS[#]}, when quoted, is treated specially as a sequence of arguments, each of which is quoted individually to preserve any whitespace or special characters in the individual elements. If arr=("a b" c d), then echo "${arr[#]}" is the same as
echo "a b" "c" "d"
rather than
echo "a b c d"
This will not work in a shell that doesn't support arrays, but then, arrays were invented because there wasn't a safe way (that is, without using eval) to handle this use case without them.
This:
rsync "$OPT1 $SRC $TRG"
passes all your intended arguments lumped together as one argument, which rsync doesn't know how to deal with.
Try this instead:
rsync ${OPT1} ${SRC} ${TRG}
The approach suggested by #chepner didn't work on my Mac OS X (10.9.4), but eval did.
eval rsync "$OPT1 $SRC $TRG"

How to properly pass run-time determined command line switches that include *'s in bash?

I am writing a simple script that rsync's a remote site to my local computer, and dynamically generates --exclude=dir flags depending on what option is specified on the command line.
#!/bin/bash -x
source="someone#somewhere.org:~/public_html/live/"
destination="wordpress/"
exclude_flags='--exclude=cache/* '
if [ "$1" == "skeleton" ] ; then
exclude_flags+='--exclude=image-files/* '
fi
rsync --archive --compress --delete $exclude_flags -e ssh $source $destination
I'm running into trouble when I try to interpolate the $exclude_flags variable on the last line. Since the variable has spaces in it, bash is automatically inserting single quotes before and after the interpolation. Here is the command which bash tried to execute (the relevant output of /bin/bash +x):
+ /usr/bin/rsync --archive --compress --delete '--exclude=cache/*' '--exclude=image-files/*' -e /usr/bin/ssh someone#somewhere.org:~/public_html/live/ wordpress/
As you can see, bash has inserted a bunch of single quotes around the individual tokens of $exclude_flags, which is causing rsync to choke.
I have tried:
What I have listed above.
Putting it in double quotes ... "$exclude_flags" .... This almost fixes the problem, but not quite. The single quotes only appear around the full content of $exclude_flags, rather than around each token.
Making $exclude_flags an array, and then interpolating it using ${exclude_flags[#]}. This gives the same output as #2.
Wrapping the whole rsync line in back-tick quotes. This gives the same output as #1.
Any ideas? This seems like a really simple and common problem in bash, so I'm sure that I'm doing something wrong, but google didn't help at all.
Thank you.
The proper way to store multiple command-line options in a variable in bash is to use an array:
source="someone#somewhere.org:~/public_html/live/"
destination="wordpress/"
options=( '--exclude=cache/*' )
if [[ "$1" == "skeleton" ]] ; then
options+=( '--exclude=image-files/*' )
fi
rsync --archive --compress --delete "${exclude_flags[#]}" -e ssh "$source" "$destination"

bash: rsync with options as variable

I am writing bash script, which in some part will rsync files over ssh. Unfortunately I am facing problem with keeping rsync options as variable. Please take a look below:
# variables
directory="/etc"
backupDirectory="/backup"
incrementalBackup="/incremental"
options="-a -e 'ssh -p 10022' -b --backup-dir=$incrementalBackup --delete"
# rsync
rsync $options user#server:$directory $backupDirectory
Unfortunately above script fails with rsync error:
Unexpected remote arg: user#server:/etc
rsync error: syntax or usage error (code 1) at main.c(1201) [sender=3.0.6]
What I saw during script debugging is the fact, that ssh options ('ssh -p 10022') are treated as rsync options.
The question is how to pass correctly those additional ssh settings into rsync?
Thanks in advance for a tip.
Use an array; it's why they were added to bash:
# variables
directory="/etc"
backupDirectory="/backup"
incrementalBackup="/incremental"
options=(-a -e 'ssh -p 10022' -b --backup-dir="$incrementalBackup" --delete)
# rsync
rsync "${options[#]}" user#server:"$directory" "$backupDirectory"
eval is not a safe option to use; it isn't limited to just evaluating the quotations you intend it to, but will evaluate any code. It might work for your current situation, but changes to the value of options might bring unforeseen consequences, and it's generally a bad idea to get into the habit of using eval when it isn't necessary.
Use eval. Try:
eval rsync $options user#server:$directory $backupDirectory

Resources