I would like to use custom identity file when using rsync, but only if the file exists, otherwise I don't want to bother with custom ssh command for rsync. I am having problems with quotes. See examples.
Desired command if identity file exists
rsync -e "ssh -i '/tmp/id_rsa'" /tmp/dir/ u#h:/tmp/dir
Desired command if identity file does not exist
rsync /tmp/dir/ u#h:/tmp/dir
I wanted to create a variable that would contain -e "ssh -i '/tmp/id_rsa'" and use it as follows
rsync ${identityArg} /tmp/dir/ u#h:/tmp/dir
This variable would be either empty or contain desired ssh command.
An example way I fill the variable (I have tried many ways)
IDENTITY_FILE="/tmp/id_rsa"
if [ -f "${IDENTITY_FILE}" ]; then
identityArg="-e 'ssh -i \"${IDENTITY_FILE}\"'"
fi
The problem is that quotes are always wrong in the command and I end up with commands similar to these ones (set -x is set in the script and this is the output)
rsync -e '\ssh' -i '"/tmp/id_rsa"'\''' /tmp/dir/ u#h:/tmp/dir
There is something I do not get about quotation in bash. If you have any good resource about usage of single and double quotes in bash script I would like to read it.
You want to add two positional parameters: -e and ssh -i '/tmp/id_rsa', where /tmp/id_rsa is an expanded variable. You should use an array for this:
args=(/tmp/dir/ u#h:/tmp/dir)
idfile=/tmp/id_rsa
# Let [[ ... ]] do the quoting
if [[ -f $idfile ]]; then
# Prepend two parameters to args array
args=(-e "ssh -i '$idfile'" "${args[#]}")
fi
rsync "${args[#]}"
I'm not convinced the inner single quotes are necessary for ssh -i, but this expands to exactly the commands shown in the question.
Try like this
id=/tmp/id_rsa
[[ -e $id ]] && o1='-e' o2="ssh -i '$id'"
echo $o1 "$o2" /tmp/dir/ u#h:/tmp/dir
Trying to properly escape quotes is tricky. Better to try to leverage existing constructs. Few alternatives, depending on the situations
If the name of the identity file only contain simple characters (no spaces, wildcard, etc.) consider not wrapping it in quotes. In this case, you can
IDENTITY_FILE="/tmp/id_rsa"
if [ -f "${IDENTITY_FILE}" ]; then
identityArg="-e 'ssh -i ${IDENTITY_FILE}'"
fi
...
rsync $identityArg ...
Another option is to always pass in the command (ssh or 'ssh -I ...'). This will automatically take care for special characters in the identity file.
IDENTITY_FILE="/tmp/id_rsa"
if [ -f "${IDENTITY_FILE}" ]; then
identityArg="-i '${IDENTITY_FILE}'"
fi
rsync -e "ssh $identityArg" ...
Third alternative is to use array to create the arguments to rsync, and let the shell escape the characters as needed. This will allow any character in the identity file.
IDENTITY_FILE="/tmp/id_rsa"
if [ -f "${IDENTITY_FILE}" ]; then
identityArg=(-e "ssh -i '${IDENTITY_FILE}'")
fi
rsync "${identityArg[#]}" ...
Related
Question
How to expand a shell variable, single quote it, then pass it to sudo?
Attempt
This does not work.
key="some string including special characters"
file='/home/ansible/.ssh/authorized_keys'
sudo -i -u ansible grep -q -E "'"${key}"'" $file
It is not working.
sudo -i -u ansible cat /home/ansible/.ssh/authorized_keys
hoge
h oge
hoge
key='h oge'
sudo -i -u ansible grep -E "'"$key"'" $file
grep: oge': No such file or directory
I never thought I'd say this, but Use Fewer Quotes (trademark pending). This should work fine:
sudo -i -u ansible grep -q -E "$key" "$file"
Basically, you only have to provide syntactic quotes for the literal code (unless you are doing something eval).
If you have characters in key which have a special meaning in a regular expression that is an entirely different problem. In that case use grep -q -F "$key" "$file" to search for a literal string rather than an extended regular expression.
RSYNC="rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete "
# Files
$RSYNC deploy#ip:/var/www/path1 /var/www/path1
$RSYNC deploy#ip:/var/www/path2 /var/www/path2
I'd like to introduce this RSYNC Variable to be more compact, but it throws an error:
Unexpected remote arg: deploy#ip:/var/www/path1
If i use only rsync inside the doublequotes, it works fine. For the sake of readability, i'd keep them separate command invocations.
I agree that eval is dangerous. In addition to the array approach #Eugeniu Rosca suggested, you could also use a shell function:
my_rsync() {
rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete "$#"
}
my_rsync deploy#ip:/var/www/path1 /var/www/path1
my_rsync deploy#ip:/var/www/path2 /var/www/path2
BTW, you should read BashFAQ #50: I'm trying to put a command in a variable, but the complex cases always fail!.
If you want to store the command string into a variable and evaluate it later, you can use eval or an alternative safer technique:
#!/bin/bash
# Store the command into an array
RSYNC=(rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete)
# Run the command
"${RSYNC[#]}" deploy#ip:/var/www/path1 /var/www/path1
Why eval should be avoided
After you assign a string to a variable and then submit it to shell again it is tokenized differently. Consider the following script
VAR="/bin/ls 'Test1 Test2'"
$VAR
It will throw two errors:
/bin/ls: cannot access 'Test1
/bin/ls: cannot access Test2'
You guessed it right, the apostrophes are no longer affecting tokenization. Instead they are treated as normal characters when the commandline arguments are fed to /bin/ls
While eval is rightly considered evil, it is by far the easiest way to get a script running. It stitches all the arguments into one line and applies the tokenizing procedure again.
RSYNC="rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete "
# Files
eval $RSYNC deploy#ip:/var/www/path1 /var/www/path1
eval $RSYNC deploy#ip:/var/www/path2 /var/www/path2
I am writing a simple script that rsync's a remote site to my local computer, and dynamically generates --exclude=dir flags depending on what option is specified on the command line.
#!/bin/bash -x
source="someone#somewhere.org:~/public_html/live/"
destination="wordpress/"
exclude_flags='--exclude=cache/* '
if [ "$1" == "skeleton" ] ; then
exclude_flags+='--exclude=image-files/* '
fi
rsync --archive --compress --delete $exclude_flags -e ssh $source $destination
I'm running into trouble when I try to interpolate the $exclude_flags variable on the last line. Since the variable has spaces in it, bash is automatically inserting single quotes before and after the interpolation. Here is the command which bash tried to execute (the relevant output of /bin/bash +x):
+ /usr/bin/rsync --archive --compress --delete '--exclude=cache/*' '--exclude=image-files/*' -e /usr/bin/ssh someone#somewhere.org:~/public_html/live/ wordpress/
As you can see, bash has inserted a bunch of single quotes around the individual tokens of $exclude_flags, which is causing rsync to choke.
I have tried:
What I have listed above.
Putting it in double quotes ... "$exclude_flags" .... This almost fixes the problem, but not quite. The single quotes only appear around the full content of $exclude_flags, rather than around each token.
Making $exclude_flags an array, and then interpolating it using ${exclude_flags[#]}. This gives the same output as #2.
Wrapping the whole rsync line in back-tick quotes. This gives the same output as #1.
Any ideas? This seems like a really simple and common problem in bash, so I'm sure that I'm doing something wrong, but google didn't help at all.
Thank you.
The proper way to store multiple command-line options in a variable in bash is to use an array:
source="someone#somewhere.org:~/public_html/live/"
destination="wordpress/"
options=( '--exclude=cache/*' )
if [[ "$1" == "skeleton" ]] ; then
options+=( '--exclude=image-files/*' )
fi
rsync --archive --compress --delete "${exclude_flags[#]}" -e ssh "$source" "$destination"
Trying to copy a remote file to my local system using scp in bash
I've obtained the filename that i want and assigned to variable, $lastModifiedFile,
but the problem is it contains spaces in the filename.
To use this variable with scp the spaces need to be escaped with backslashes.
Is there an easy way to format this variable and insert the correct escape character where necessary i.e on spaces?
#!/bin/bash
lastModifiedFile=$(sshpass -p 'passw0rd' ssh user#server 'ls -tr /path/output*| tail -n 1')
echo "$lastModifiedFile"
sshpass -p 'passw0rd' scp user#server:"$lastModifiedFile" /root/
This is the script output ..
[user#host ~]# ./script.sh
/path/outputSat Mar 09 151905 GMT 2013.html
scp: /path/outputSat: No such file or directory
scp: Mar: No such file or directory
scp: 09: No such file or directory
scp: 151905: No such file or directory
scp: GMT: No such file or directory
scp: 2013.html: No such file or directory
I'm looking for something like below, or even a simpler solution? ..
escapedFilename=""
for letter in $lastModifiedFile
if $letter == " "
$escapedFilename += "\ "
else
$escapedFilename += $letter
With a bit of leaning toothpick syndrome:
param=user#server:${lastModifiedFile// /\\ /}
sshpass -p 'passw0rd' scp "$param" /root/
EDIT: It seems scp does not like me. I needed an additional level of variable in testing ... :)
EDIT 2: According to OP's feedback the exact solution appears to consist of using ${lastModifiedFile// /\\ \\}
I just hope there are no other characters than space that need escaping in some other filenames :)
Use single quotes around the filename passed to the remote system so that it is not subject to word splitting.
lastModifiedFile=$(sshpass -p 'passw0rd' ssh user#server 'ls -tr /path/output*| tail -n 1')
echo "$lastModifiedFile"
sshpass -p 'passw0rd' scp user#server:"'$lastModifiedFile'" /root/
or
sshpass -p 'passw0rd' scp "user#server:'$lastModifiedFile'" /root/
Just do it like this:
sshpass -p 'passw0rd' scp 'user#server:$lastModifiedFile' /root/
Here are a couple of methods that should handle almost anything (not just spaces) in the filename. First, bash's printf builtin has a %q format that adds quotes/escapes/whatever to the string:
sshpass -p 'passw0rd' scp user#server:"$(printf %q "$lastModifiedFile")" /root/
Note, however, that this quotes/escapes/etc it suitably for interpretation by bash. If the remote computer's default shell is something else, this may not work in all cases.
Option two is simpler in principle (but a bit messy in practice), and should be compatible with more remote shells. Here, I enclose the filename in single-quotes, which should work for anything other than single-quotes within the filename. For those, I substitute '\'' (which ends the single-quoted string, adds an escaped single-quote, then restarts the single-quoted string):
repl="'\''" # Have to store this in a variable to work around a bash parsing oddity
sshpass -p 'passw0rd' scp user#server:"'${lastModifiedFile//\'/$repl}'" /root/
I am trying to upload multiple files from one folder to a ftp site and wrote this script:
#!/bin/bash
for i in '/dir/*'
do
if [-f /dir/$i]; then
HOST='x.x.x.x'
USER='username'
PASSWD='password'
DIR=archives
File=$i
ftp -n $HOST << END_SCRIPT
quote USER $USER
quote PASS $PASSWD
ascii
put $FILE
quit
END_SCRIPT
fi
It is giving me following error when I try to execute:
username#host:~/Documents/Python$ ./script.sh
./script.sh: line 22: syntax error: unexpected end of file
I can't seem to get this to work. Any help is much appreciated.
Thanks,
Mayank
It's complaining because your for loop does not have a done marker to indicate the end of the loop. You also need more spaces in your if:
if [ -f "$i" ]; then
Recall that [ is actually a command, and it won't be recognized if it doesn't appear as such.
And... if you single quote your glob (at the for) like that, it won't be expanded. No quotes there, but double quotes when using $i. You probably also don't want to include the /dir/ part when you use $i as it's included in your glob.
If I'm not mistaken, ncftp can take wildcard arguments:
ncftpput -u username -p password x.x.x.x archives /dir/*
If you don't already have it installed, it's likely available in the standard repo for your OS.
First, the literal, fixing-your-script answer:
#!/bin/bash
# no reason to set variables that don't change inside the loop
host='x.x.x.x'
user='username'
password='password'
dir=archives
for i in /dir/*; do # no quotes if you want the wildcard to be expanded!
if [ -f "$i" ]; then # need double quotes and whitespace here!
file=$i
ftp -n "$host" <<END_SCRIPT
quote USER $user
quote PASS $password
ascii
put $file $dir/$file
quit
END_SCRIPT
fi
done
Next, the easy way:
lftp -e 'mput -a *.i' -u "$user,$password" "ftp://$host/"
(yes, lftp expands the wildcard internally, rather than expecting this to be done by the outer shell).
First of all my apologies in not making myself clear in the question. My actual task was to copy a file from local folder to a SFTP site and then move the file to an archive folder. Since the SFTP is hosted by a vendor I cannot use the key sharing (vendor limitation. Also, SCP will require password entering if used in a shell script so I have to use SSHPASS. SSHPASS is in the Ubuntu repo however for CentOS it needs to be installed from here
Current thread and How to run the sftp command with a password from Bash script? did gave me better understanding on how to write the script and I will share my solution here:
#!/bin/bash
#!/usr/bin
for i in /dir/*; do
if [ -f "$i" ]; then
file=$i
export SSHPASS=password
sshpass -e sftp -oBatchMode=no -b - user#ftp.com << !
cd foldername/foldername
put $file
bye
!
mv $file /somedir/test
fi
done
Thanks everyone for all the responses!
--Mayank