This question already has answers here:
Variables as commands in Bash scripts
(5 answers)
Closed 6 years ago.
I'm using a bash script to sync a web folder using rsync. I'm using an array of folders and files to exclude, and I'm having problems escaping these items for the exclude list...
my exclude list is defined like so...
SYNC_EXCLUSIONS=(
'/exclude_folder_1'
'/exclude_folder_2'
'.git*'
'.svn'
)
Then I build my exclusion string like so...
exclusions='';
for e in "${SYNC_EXCLUSIONS[#]}"
do
exclusions+=" --exclude='$e'";
done
Then finally I execute my rsync...
rsync --recursive --delete $exclusions "$DEPLOYMENT_WORK_DIR/" "$DEPLOYMENT_ROOT/"
If I echo the command it looks perfect, and if I copy and execute it at the prompt it works correctly. However when run from the script the exclusions are ignored.
I've figured out that it will work if I remove the single quotes from around each excluded item, like so...
exclusions+=" --exclude=$e";
I'd prefer to not do that though, just in case I need to exclude folders with spaces or special characters.
Is there some way I can get this to work from the script while retaining quotes around the excluded items? I've tried all sorts of combinations of quotes and backslashes etc. and nothing I've tried works.
You can't build a string for this at all -- see BashFAQ #50 for an extensive discussion of why. Build an array.
exclusions=( )
for e in "${SYNC_EXCLUSIONS[#]}"; do
exclusions+=( --exclude="$e" )
done
rsync --recursive --delete "${exclusions[#]}" "$DEPLOYMENT_WORK_DIR/" "$DEPLOYMENT_ROOT/"
...well, can't build a string at all, unless you're going to execute it with eval. Doing that in a manner that isn't prone to shell injection vulnerabilities takes care, however:
printf -v exclusions_str '--exclude=%q ' "${SYNC_EXCLUSIONS[#]}"
printf -v rsync_cmd 'rsync --recursive --delete %s %q %q' \
"$exclusions_str" "$DEPLOYMENT_WORK_DIR/" "$DEPLOYMENT_ROOT/"
eval "$rsync_cmd"
Related
This question already has answers here:
Setting an argument with bash [duplicate]
(2 answers)
Closed 6 years ago.
I wrote a bash script with the following:
SRC="dist_serv:$HOME/www/"
DEST="$HOME/www/"
OPTIONS="--exclude 'file.php'"
rsync -Cavz --delete $OPTIONS $SRC $DEST
rsync fails and I can't figure out why, although it seems to be related to the $OPTIONS variable (it works when I remove it). I tried escaping the space with a backslash (among many other things) but that didn't work.
The error message is :
rsync: mkdir "/home/xxx/~/public_html/" failed: No such file or directory (2)
I tried quoting the variable, which throws another error ("unknown option" on my variable $OPTIONS):
rsync: --exclude 'xxx': unknown option
rsync error: syntax or usage error (code 1) at main.c(1422) [client=3.0.6]
You shouldn't put $ in front of the variable names when assigning values to them. SRC is a variable, $SRC is the value that it expands to.
Additionally, ~ is not expanded to the path of your home directory when you put it in quotes. It is generally better to use $HOME in scripts as this variable behaves like a variable, which ~ doesn't do.
Always quote variable expansions:
rsync -Cavz --delete "$OPTIONS" "$SRC" "$DEST"
unless there is some reason not to (there very seldom is). The shell will perform word splitting on them otherwise.
User #Fred points out that you can't use double quotes around $OPTIONS (in in comments below), but it should be ok if you use OPTIONS='--exclude="file.php"' (note the =).
One technique which I find invaluable is using positional parameters to make it easy to work with list of options.
When you put options inside a variable (such as your OPTIONS variable), you need to find a way to include quotes inside the value, and omit quotes when referencing the variable. It works, but you are always one typo away from a difficult to debug failure.
Instead, try the following.
set -- -Cavz --delete
set -- "$#" --exclude "file.php"
set -- "$#" "dist_serv:~/www/"
set -- "$#" "~/www/"
rsync "$#"
Of course, in this case, everything could be on the same line, but in many cases there will be conditional expressions so that, for instance, you can omit a given option, or select difference files to work with. The nice thing is, you always use the same quoting you would use on a single command line, all thanks to the magic of "$#" that avoids having to reference (or quote) any specific variable.
If actual positional parameters get in the way, you can put them in variables, or create a function to isolate a context that avoids touching them where they matter.
I use this trick all the time, and I have stopped pulling my hair out due to quoting causing problems inside values I pass as parameter to commands.
A similar result can be achieved by using an array.
declare -a ARGUMENTS=()
ARGUMENTS=(-Cavz --delete )
ARGUMENTS+=(--exclude "file.php")
ARGUMENTS+=("dist_serv:~/www/")
ARGUMENTS+=("~/www/")
rsync "${ARGUMENTS[#]}"
I'm trying to do an rsync like this in a bash script;
rsync -e "ssh ${flags_ssh}" -avRz --rsync-path="sudo rsync" $direcNew $(eval echo ${user_name})#$(eval echo ${instance_address}):$(eval echo ${mountdir}`)
However, when I run this using bash -x like this:
bash -x ./myscript
I get that it's trying to run that command except with the option rsync-path looking like
'--rsync-path=sudo rsync'
How do I prevent this? I need the double quotes to stay and the single quotes to go away, I don't know why it's happening, and I've tried endless combinations of eval and backslashes with no success.
You don't need to prevent this. What's happening is that when the shell parses --rsync-path="sudo rsync", it removes the quotes (after they have the intended effect of having the space treated as part of the argument, rather than a separator between arguments). Then, when it sees it's in -x mode, it comes up with a representation that would have led to the space being treated that way, and prints that. It could print any equivalent representation, including (but not limited to) any of these:
--rsync-path="sudo rsync"
"--rsync-path=sudo rsync"
--rsync-path=sudo\ rsync
--rsync-path=sudo" "rsync
'--rsync-path=sudo rsync'
$'--rsync-path=sudo rsync'
...etc
The fact that it picked a different representation than you did is not important, because these are all fully equivalent -- they all result in exactly the same thing being passed as an argument to rsync, so you don't need to worry about it.
You also intrinsically can't "fix" it, because by the time the shell prints its interpretation of your command, it's already forgotten which representation you happened to use -- it only knows the resulting argument that's going to be passed to rsync. In order to get it to print something else, you'd have to be passing a different actual argument to rsync, and that would break your rsync command.
Originally I would like to sync directory (with all files and subdirectories) given in parameter in bash script.
I found this post: How can I recursively copy a directory into another and replace only the files that have not changed? which explains how to use rsync in similar case.
My bash script is quite simple and listed below:
#!/bin/bash
echo -e "Type the project to be deployed: \c "
read project
echo -e "* Deploying: $project *"
echo -e "Sync: /var/repo/released/$project"
echo -e " /var/www/released/$project"
rsync -pr /var/repo/released/$project /var/www/released/$project
As a result it copies everything within /released (there are many directories in there, let's say -projects-).
I would like to copy (sync) only project given in parameter.
Could you please advice how to do this?
When you call the script without an argument (which most likely is what you're doing since you interactively read the project name into the variable $project), the positional parameter $1 remains empty. Therefore the script will rsync the entire content of /var/repo/released/.
You need to replace $1 with $project in your script. Also, I'd recommend to put double quotes around the paths to avoid problems due to spaces in a directory name.
rsync -pr "/var/repo/released/$project" "/var/www/released/$project"
I've run into this a few times and I'm curious about automating it. If I want to move a bunch of files matching a pattern to a slightly different pattern, how would I do it?
More specifically, If I want to git mv the following files
fileA.css.scss
fileB.css.scss
...
to
fileA.scss
fileA.scss
...
How would you do it?
I would write something like:
for file in *.css.scss ; do
mv "$file" "${file%.css.scss}.scss"
done
(Note: I'm not sure of the right arguments for git mv, so I just demonstrated using mv, I hope that's O.K.)
For information on the ${parameter%word} notation, see ยง3.5.3 Shell Parameter Expansion in the Bash Reference Manual.
Been searching for hours and still cant figure this out :(
Anyway, im creating a script that will automatically rsync about 40 different directories to 40 other directories on another server. if you want to see the entire script you can view it here: http://pastebin.com/Nt3KKvx9
But the important bit is the for loop where i run rsync
for ((i=0; i<${#websys[#]}; i++))
do
localpath="/nusdata/staff/NUS/NUS/Systems/${kiskasys[$i]}"
remotepath="/home/www/html/nusalaska.com/html/systems/${websys[$i]}"
rsync -rlptnvz -s "$localpath" -e "ssh -p 50014" "nusak#webserver:$remotepath/"
done
The problem is that the array "kiskasys" has many directory names that have spaces in them (Example: '101 greenbrook').
I have tried making the array variables have single quotes around them, double quotes around them, escaped spaces like '\ ', and combinations of all three. I have also tried putting the $localpath in quotes, not in quotes, etc. etc.
I guess im just confused on how the -s (--protect-args) deals with the spaces and how I can get it to work in my situation.
The error output always looks something like the following:
rsync: change_dir "/nusdata/staff/NUS/NUS/101" failed: No such file or directory (2)
or
rsync: change_dir "/nusdata/staff/NUS/NUS/'101 greenbrook'" failed: No such file or directory (2)
Any help is appreciated!
Thanks
Found My Problem
In copying my code from my script to this page I accidently copied it wrong... however in copying it wrong, what i posted above was perfectly good code that works fine haha.
so what i posted above was the solution to my own problem.
The original had single quotes in the localpath variable like so:
localpath="'/nusdata/staff/NUS/NUS/Systems/${kiskasys[$i]}'"
and the single quotes was the problem. And for everyone's benefit here is an example output
echo ${kiskasys[1]}
#output would look like this:
101 greenbrook
Basically there are no special escape characters etc.
For what it's worth, I'm not able to replicate your problem. If I set localpath and remotepath to a directory with spaces:
localpath="/home/lars/tmp/so/dir1/a directory"
remotepath="/home/lars/tmp/so/dir2/a directory"
And then run your rsync command (modified slightly for my environment):
rsync -rlptvz -s "$localpath/" -e "ssh" "localhost:$remotepath/"
It Just Works. Here's the content of dir1:
dir1/a directory/file1
dir1/a directory/file3
dir1/a directory/file2
And after running rsync, dir2 looks like this:
dir2/a directory/file1
dir2/a directory/file3
dir2/a directory/file2
Can you show a specific command line that results in the errors you're seeing above? Maybe run the script with the -x flag and show exactly how localpath and remotepath are set.