When I run my script it fails:
# sh -x ./rsync.sh
...
+ rsync --safe-links --password-file=/etc/rsync.secret -rtvun --include='*.log*.gz' --filter='-! */' --prune-empty-dirs /some/path foo#hostname::foo
Unknown filter rule: `'-!'
rsync error: syntax or usage error (code 1) at exclude.c(904) [client=3.1.1]
but when I copy&paste the rsync --safe-links .... line and execute it it works fine.
rsync.sh:
#!/bin/bash
rsync_opts="--safe-links --password-file=/etc/rsync.secret -rtvu"
while read path hostname volume extra_opts; do
rsync ${rsync_opts} ${extra_opts} ${path} ${volume}#${hostname}::${volume}
done < /etc/rsync.paths
rsync.paths:
/some/path/ hostname foo --include='*.log*.gz' --filter='-! */' --prune-empty-dirs
Does anyone knows why it fails from the script and works fine when I run the command manually ? How can I fix it ?
UPDATE:
It works fine when I use eval "rsync ..." but still I don't know why it doesn't work withou it :/
Your problem is that your script reads the text --filter='-! */' --prune-empty-dirs into the variable $extra_opts, which is passed literally to the command, including the apostrophes. So it is split into the following arguments: --filter='-!, */' and --prune-empty-dirs. When you eval it though, then the whole line is passed to the shell interpreter which interprets the apostrophes the way you want, that's why it works.
Also, since your script uses bash, you should debug it with bash -x instead of sh -x.
Related
I wanted to make an alias for launching a vim session with all the c/header/makefiles, etc loaded into the buffer.
shopt -s extglob
alias vimc="files=$(ls -A *.?(c|h|mk|[1-9]) .gitconfig [mM]akefile 2>/dev/null); [[ -z $files ]] || vim $files"
When I run the command enclosed within the quotations from the shell, it works but when run as the alias itself, it does not. Running vimc, causes vim to launch only in the first matched file(which happens to be the Makefile) and the other files(names) are executed as commands for some reason(of course unsuccessfully). I tried fiddling around and it seems that the command substitution introduces the problem. Because running only the ls produces expected output.
I cannot use xargs with vim because it breaks the terminal display.
Can anyone explain what might be causing this ?
Here is some output:
$ ls
Makefile readme main.1 main.c header.h config.mk
$ vimc
main.1: command not found
main.c: command not found
.gitignore: command not found
header.h: command not found
config.mk: command not found
On an related note, would it be possible to do what I intend to do above in a "single line", i.e without storing it into a variable files and checking to see if it is empty, using only the output stream from ls?
I have an rsync command in my csh script like this:
#! /bin/csh -f
set source_dir = "blahDir/blahBlahDir"
set dest_dir = "foo/anotherFoo"
rsync -av --exclude=*.csv ${source_dir} ${dest_dir}
When I run this I get the following error:
rsync: No match.
If I remove the --exclude option it works. I wrote the equivalent script in bash and that works as expected
#/bin/bash -f
source_dir="blahDir/blahBlahDir"
dest_dir="foo/anotherFoo"
rsync -av --exclude=*.csv ${source_dir} ${dest_dir}
The problem is that this has to be done in csh only. Any ideas on how I can get his to work?
It's because csh is trying to expand --exclude=*.csv into a filename, and complaining because it cannot find a file matching that pattern.
You can get around this by enclosing the option in quotes:
rsynv -rv '--exclude=*.csv' ...
or escaping the asterisk:
rsynv -rv --exclude=\*.csv ...
This is a consequence of the way csh and bash differ in their default treatment of arguments with wildcards that don't match a file. csh will complain while bash will simply leave it alone.
You may think bash has chosen the better way but that's not necessarily so, as shown in the following transcript where you have a file matching the argument:
pax> touch -- '--file=xyzzy.csv' ; ls -- *.csv
--file=xyzzy.csv
pax> echo --file=*.csv
--file=xyzzy.csv
You can see there that the bash shell expands the argument rather than giving it to the program as is. Both sides have their pros and cons.
RSYNC="rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete "
# Files
$RSYNC deploy#ip:/var/www/path1 /var/www/path1
$RSYNC deploy#ip:/var/www/path2 /var/www/path2
I'd like to introduce this RSYNC Variable to be more compact, but it throws an error:
Unexpected remote arg: deploy#ip:/var/www/path1
If i use only rsync inside the doublequotes, it works fine. For the sake of readability, i'd keep them separate command invocations.
I agree that eval is dangerous. In addition to the array approach #Eugeniu Rosca suggested, you could also use a shell function:
my_rsync() {
rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete "$#"
}
my_rsync deploy#ip:/var/www/path1 /var/www/path1
my_rsync deploy#ip:/var/www/path2 /var/www/path2
BTW, you should read BashFAQ #50: I'm trying to put a command in a variable, but the complex cases always fail!.
If you want to store the command string into a variable and evaluate it later, you can use eval or an alternative safer technique:
#!/bin/bash
# Store the command into an array
RSYNC=(rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete)
# Run the command
"${RSYNC[#]}" deploy#ip:/var/www/path1 /var/www/path1
Why eval should be avoided
After you assign a string to a variable and then submit it to shell again it is tokenized differently. Consider the following script
VAR="/bin/ls 'Test1 Test2'"
$VAR
It will throw two errors:
/bin/ls: cannot access 'Test1
/bin/ls: cannot access Test2'
You guessed it right, the apostrophes are no longer affecting tokenization. Instead they are treated as normal characters when the commandline arguments are fed to /bin/ls
While eval is rightly considered evil, it is by far the easiest way to get a script running. It stitches all the arguments into one line and applies the tokenizing procedure again.
RSYNC="rsync -avzhe 'ssh -i /path/to/deploy_keys/id_rsa' --delete "
# Files
eval $RSYNC deploy#ip:/var/www/path1 /var/www/path1
eval $RSYNC deploy#ip:/var/www/path2 /var/www/path2
I am writing bash script, which in some part will rsync files over ssh. Unfortunately I am facing problem with keeping rsync options as variable. Please take a look below:
# variables
directory="/etc"
backupDirectory="/backup"
incrementalBackup="/incremental"
options="-a -e 'ssh -p 10022' -b --backup-dir=$incrementalBackup --delete"
# rsync
rsync $options user#server:$directory $backupDirectory
Unfortunately above script fails with rsync error:
Unexpected remote arg: user#server:/etc
rsync error: syntax or usage error (code 1) at main.c(1201) [sender=3.0.6]
What I saw during script debugging is the fact, that ssh options ('ssh -p 10022') are treated as rsync options.
The question is how to pass correctly those additional ssh settings into rsync?
Thanks in advance for a tip.
Use an array; it's why they were added to bash:
# variables
directory="/etc"
backupDirectory="/backup"
incrementalBackup="/incremental"
options=(-a -e 'ssh -p 10022' -b --backup-dir="$incrementalBackup" --delete)
# rsync
rsync "${options[#]}" user#server:"$directory" "$backupDirectory"
eval is not a safe option to use; it isn't limited to just evaluating the quotations you intend it to, but will evaluate any code. It might work for your current situation, but changes to the value of options might bring unforeseen consequences, and it's generally a bad idea to get into the habit of using eval when it isn't necessary.
Use eval. Try:
eval rsync $options user#server:$directory $backupDirectory
In my bash shell scripting,
rsync -a --delete -e "ssh -i /keypath" /source_dir/ username#remotehost:/dest_dir/
whatever combinations [eval, double to single quotes] of above gives below err,
Missing trailing-' in remote-shell command.
rsync error: syntax or usage error (code 1) at ../main.c(335) [sender=2.6.8]
But the same command in directly working in command prompt.
Could anyone help in what im missing here
Thanks
Your problem is with embedded quotes in the bash script.
Bash FAQ #50
See the following solution (and upvote it if you find it useful, please):
https://superuser.com/questions/354361/rsync-complaining-about-missing-trailing-in-a-bash-script
Not sure what your problem is but for me most of weird rsync errors are caused by the .bashrc or .bash_profile (or other shell rc files) outputting something when logging in. For now I usually write the following statement as the first line in my .bashrc and .bash_profile:
# return if it's not an interactive shell
[[ $- == *i* ]] || return 0
An easy solution that worked for me is using single quotes instead of double quotes, i.e.
rsync -a --delete -e 'ssh -i /keypath' /source_dir/ username#remotehost:/dest_dir/