I'm trying to call arguments for running a script with the rsync command. I've tried various forms and this is the most "simple", however I continue to get an error.
(bash.sh)
#! /bin/bash
root_dir = $1
target_dir = $2
rsync -avh -P --stats $root_dir $target_dir
echo "Files Transferred!"
ErrorMessage
test.sh: line 2: root_dir: command not found
test.sh: line 3: target_dir: command not found
Command Line
sh bash.sh ./path_root_dir ./path_target_dir
Using a variable with a space, without using speech marks or a quote on each side can cause an error to occur, especially when the variable is used and there was a space in the value.
Another way you could try is to use an array, the options variable is where the array is, and then it is called in the rsync command below,
directory="/etc"
backupDirectory="/backup"
incrementalBackup="/incremental"
options=(-a -e 'ssh -p 10022' -b --backup-dir="$incrementalBackup" --delete)
# rsync
rsync "${options[#]}" user#server:"$directory" "$backupDirectory"
Related
I have written following bash script:
#!/bin/sh
echo "Number of command line arguments : $#"
if [ $# == 0 ]; then
echo "Your command line contains no arguments"
declare -a arr=("xx.xx.xx.xx" "yy.yy.yy.yy")
else
declare -a arr=($1)
fi
for i in "${arr[#]}"
do
URL="https://"$i":8443"
echo "URL is $URL"
wget --no-check-certificate $URL/heapdump
done
It keeps failing with line 16: wget: command not found
I have found some related posts but could not figure out how to fix this. Thanks.
If you type a command (which is is not an internal command or a shell function), bash searches the directories mentioned in the variable PATH for a file of this name (wget in your case), which has the executable bit set for the user running the script. In your case, no suitable wget has been found.
Since it is unlikely that you do have a wget in your PATH without x-bit set, the most likely cause is that the PATH is lacking the directory where your wget lives.
You have two options: Extend the PATH, or explicitly prefix the wget line in your script with the correct path, i.e.
/here/is/my/wget --no-check-certificate $URL/heapdump
For getting path of executable files such as wget please use ‘which’ cmd:
which wget
Output:
/full/path/wget
When I run my script it fails:
# sh -x ./rsync.sh
...
+ rsync --safe-links --password-file=/etc/rsync.secret -rtvun --include='*.log*.gz' --filter='-! */' --prune-empty-dirs /some/path foo#hostname::foo
Unknown filter rule: `'-!'
rsync error: syntax or usage error (code 1) at exclude.c(904) [client=3.1.1]
but when I copy&paste the rsync --safe-links .... line and execute it it works fine.
rsync.sh:
#!/bin/bash
rsync_opts="--safe-links --password-file=/etc/rsync.secret -rtvu"
while read path hostname volume extra_opts; do
rsync ${rsync_opts} ${extra_opts} ${path} ${volume}#${hostname}::${volume}
done < /etc/rsync.paths
rsync.paths:
/some/path/ hostname foo --include='*.log*.gz' --filter='-! */' --prune-empty-dirs
Does anyone knows why it fails from the script and works fine when I run the command manually ? How can I fix it ?
UPDATE:
It works fine when I use eval "rsync ..." but still I don't know why it doesn't work withou it :/
Your problem is that your script reads the text --filter='-! */' --prune-empty-dirs into the variable $extra_opts, which is passed literally to the command, including the apostrophes. So it is split into the following arguments: --filter='-!, */' and --prune-empty-dirs. When you eval it though, then the whole line is passed to the shell interpreter which interprets the apostrophes the way you want, that's why it works.
Also, since your script uses bash, you should debug it with bash -x instead of sh -x.
RSYNC="rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete "
# Files
$RSYNC deploy#ip:/var/www/path1 /var/www/path1
$RSYNC deploy#ip:/var/www/path2 /var/www/path2
I'd like to introduce this RSYNC Variable to be more compact, but it throws an error:
Unexpected remote arg: deploy#ip:/var/www/path1
If i use only rsync inside the doublequotes, it works fine. For the sake of readability, i'd keep them separate command invocations.
I agree that eval is dangerous. In addition to the array approach #Eugeniu Rosca suggested, you could also use a shell function:
my_rsync() {
rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete "$#"
}
my_rsync deploy#ip:/var/www/path1 /var/www/path1
my_rsync deploy#ip:/var/www/path2 /var/www/path2
BTW, you should read BashFAQ #50: I'm trying to put a command in a variable, but the complex cases always fail!.
If you want to store the command string into a variable and evaluate it later, you can use eval or an alternative safer technique:
#!/bin/bash
# Store the command into an array
RSYNC=(rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete)
# Run the command
"${RSYNC[#]}" deploy#ip:/var/www/path1 /var/www/path1
Why eval should be avoided
After you assign a string to a variable and then submit it to shell again it is tokenized differently. Consider the following script
VAR="/bin/ls 'Test1 Test2'"
$VAR
It will throw two errors:
/bin/ls: cannot access 'Test1
/bin/ls: cannot access Test2'
You guessed it right, the apostrophes are no longer affecting tokenization. Instead they are treated as normal characters when the commandline arguments are fed to /bin/ls
While eval is rightly considered evil, it is by far the easiest way to get a script running. It stitches all the arguments into one line and applies the tokenizing procedure again.
RSYNC="rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete "
# Files
eval $RSYNC deploy#ip:/var/www/path1 /var/www/path1
eval $RSYNC deploy#ip:/var/www/path2 /var/www/path2
I have been working on a backup script that uses rsync to do an incremental backup.
I have tested the following rsync command manually, and it runs and completes a backup without error:
rsync -aAXv --delete --progress --link-dest=/backup/Uyuk/Uyuk-backup-part1/2014-02-24/ /mnt/backup/ /backup/Uyuk/Uyuk-backup-part1/2014-02-25/
however when I run that same command in my backup script it gives me the following error:
rsync: -aAXv --delete --progress --link-dest=/backup/Uyuk/Uyuk-backup-part1/2014-02-24/ /mnt/backup/ /backup/Uyuk/Uyuk-backup-part1/2014-02-25/: unknown option
rsync error: syntax or usage error (code 1) at main.c(1422) [client=3.0.6]
I ran bash -x on my script to figure out exactly what is sent to the console and here is what was printed:
+ rsync '-aAXv --delete --progress --link-dest=/backup/Uyuk/Uyuk-backup-part1/2014-02-24/ /mnt/backup/ /backup/Uyuk/Uyuk-backup-part1/2014-02-25/'
Does anyone see what is wrong? I cant find anything that would cause the syntax error.
EDIT:
Here is the actual code I have in the script, and this is a pretty large script so yes some variables are not defined here, but you get the idea.
mkdir -p "/backup/$HOST/$NAME/$TODAY"
#source directory
SRC="$MNT"
#link directory
LNK="/backup/$HOST/$NAME/$LAST/"
#target directory
TRG="/backup/$HOST/$NAME/$TODAY/"
#rsync options
OPT1="-aAXv --delete --progress --link-dest=$LNK"
#run the rsync command
echo "rsync $OPT1 $SRC $TRG"
rsync "$OPT1 $SRC $TRG" > /var/log/backup/backup.rsync.log 2>&1
You are passing your option list as a single argument, when it needs to be passed as a list of arguments. In general, you should use an array in bash to hold your arguments, in case any of them contain whitespace. Try the following:
mkdir -p "/backup/$HOST/$NAME/$TODAY"
#source directory
SRC="$MNT"
#link directory
LNK="/backup/$HOST/$NAME/$LAST/"
#target directory
TRG="/backup/$HOST/$NAME/$TODAY/"
#rsync options
OPTS=( "-aAXv" "--delete" "--progress" "--link-dest=$LNK" )
#run the rsync command
echo "rsync $OPT1 $SRC $TRG"
rsync "${OPTS[#]}" "$SRC" "$TRG" > /var/log/backup/backup.rsync.log 2>&1
An array expansion ${OPTS[#]}, when quoted, is treated specially as a sequence of arguments, each of which is quoted individually to preserve any whitespace or special characters in the individual elements. If arr=("a b" c d), then echo "${arr[#]}" is the same as
echo "a b" "c" "d"
rather than
echo "a b c d"
This will not work in a shell that doesn't support arrays, but then, arrays were invented because there wasn't a safe way (that is, without using eval) to handle this use case without them.
This:
rsync "$OPT1 $SRC $TRG"
passes all your intended arguments lumped together as one argument, which rsync doesn't know how to deal with.
Try this instead:
rsync ${OPT1} ${SRC} ${TRG}
The approach suggested by #chepner didn't work on my Mac OS X (10.9.4), but eval did.
eval rsync "$OPT1 $SRC $TRG"
I have a set of scripts that I use to download files via FTP and then delete them from the server.
It works as follows:
for dir in `ls /volume1/auto_downloads/sync-complete`
do
if [ "x$dir" != *"x"* ]
then
echo "DIR: $dir"
echo "Moving out of complete"
# Soft delete from server so they don't get downloaded again
ssh dan#172.19.1.15 mv -v "'/home/dan/Downloads/complete/$dir'" /home/dan/Downloads/downloaded
Now $dir could be "This is a file" which works fine.
The problem I'm having is with special characters eg:
"This is (a) file"
This is a file & stuff"
tend to error:
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `mv -v '/home/dan/Downloads/complete/This is (a) file' /home/dan/Downloads/downloaded'
I can't work out how to escape it so both the variable gets evaluated and the command gets escaped properly. I've tried various combinations of escape characters, literal quotes, normal quotes, etc
If both sides are using bash, you can escape the arguments using printf '%q ', eg:
ssh dan#172.19.1.15 "$(printf '%q ' mv -v "/home/dan/Downloads/complete/$dir" /home/dan/Downloads/downloaded)"
You need to quote the whole expression ssh user#host "command":
ssh dan#172.19.1.15 "mv -v /home/dan/Downloads/complete/$dir /home/dan/Downloads/downloaded"
I'm confused, because your code as written works for me:
> dir='foo & bar (and) baz'
> ssh host mv -v "'/home/dan/Downloads/complete/$dir'" /home/dan/Downloads/downloaded
mv: cannot stat `/home/dan/Downloads/complete/foo & bar (and) baz': No such file or directory
For debugging, use set -vx at the top of the script to see what's going on.
Will Palmer's suggestion of using printf is great but I think it makes more sense to put the literal parts in printf's format.
That way, multi-command one-liners are more intuitive to write:
ssh user#host "$(printf 'mkdir -p -- %q && cd -- "$_" && tar -zx' "$DIR")"
One can use python shlex.quote(s) to
Return a shell-escaped version of the string s
docs