This question already has answers here:
Setting an argument with bash [duplicate]
(2 answers)
Closed 6 years ago.
I wrote a bash script with the following:
SRC="dist_serv:$HOME/www/"
DEST="$HOME/www/"
OPTIONS="--exclude 'file.php'"
rsync -Cavz --delete $OPTIONS $SRC $DEST
rsync fails and I can't figure out why, although it seems to be related to the $OPTIONS variable (it works when I remove it). I tried escaping the space with a backslash (among many other things) but that didn't work.
The error message is :
rsync: mkdir "/home/xxx/~/public_html/" failed: No such file or directory (2)
I tried quoting the variable, which throws another error ("unknown option" on my variable $OPTIONS):
rsync: --exclude 'xxx': unknown option
rsync error: syntax or usage error (code 1) at main.c(1422) [client=3.0.6]
You shouldn't put $ in front of the variable names when assigning values to them. SRC is a variable, $SRC is the value that it expands to.
Additionally, ~ is not expanded to the path of your home directory when you put it in quotes. It is generally better to use $HOME in scripts as this variable behaves like a variable, which ~ doesn't do.
Always quote variable expansions:
rsync -Cavz --delete "$OPTIONS" "$SRC" "$DEST"
unless there is some reason not to (there very seldom is). The shell will perform word splitting on them otherwise.
User #Fred points out that you can't use double quotes around $OPTIONS (in in comments below), but it should be ok if you use OPTIONS='--exclude="file.php"' (note the =).
One technique which I find invaluable is using positional parameters to make it easy to work with list of options.
When you put options inside a variable (such as your OPTIONS variable), you need to find a way to include quotes inside the value, and omit quotes when referencing the variable. It works, but you are always one typo away from a difficult to debug failure.
Instead, try the following.
set -- -Cavz --delete
set -- "$#" --exclude "file.php"
set -- "$#" "dist_serv:~/www/"
set -- "$#" "~/www/"
rsync "$#"
Of course, in this case, everything could be on the same line, but in many cases there will be conditional expressions so that, for instance, you can omit a given option, or select difference files to work with. The nice thing is, you always use the same quoting you would use on a single command line, all thanks to the magic of "$#" that avoids having to reference (or quote) any specific variable.
If actual positional parameters get in the way, you can put them in variables, or create a function to isolate a context that avoids touching them where they matter.
I use this trick all the time, and I have stopped pulling my hair out due to quoting causing problems inside values I pass as parameter to commands.
A similar result can be achieved by using an array.
declare -a ARGUMENTS=()
ARGUMENTS=(-Cavz --delete )
ARGUMENTS+=(--exclude "file.php")
ARGUMENTS+=("dist_serv:~/www/")
ARGUMENTS+=("~/www/")
rsync "${ARGUMENTS[#]}"
Related
Currently trying to move all of my aliases from .bash_profile to .zshrc. However, found a problem with one of the longer aliases I use for substituting root to ubuntu when passing a command to access AWS instances.
AWS (){
cd /Users/user/aws_keys
cmd=$(echo $# | sed "s/root/ubuntu/g")
$cmd[#]
}
The error I get is AWS:5: command not found ssh -i keypair.pem ubuntu#ec1.compute.amazonaws.com
I would really appreciate any suggestions!
The basic problem is that the cmd=$(echo ... line is mashing all the arguments together into a space-delimited string, and you're depending on word-splitting to split it up into a command and its arguments. But word-splitting is usually more of a problem than anything else, so zsh doesn't do it by default. This means that rather than trying to run the command named ssh with arguments -i, keypair.pem, etc, it's treating the entire string as the command name.
The simple solution is to avoid mashing the arguments together, so you don't need word-splitting to separate them out again. You can use a modifier to the parameter expansion to replace "root" with "ubuntu". BTW, I also strongly recommend checking for error when using cd, and not proceeding if it gets an error.
So something like this:
AWS (){
cd /Users/user/aws_keys || return $?
"${#//root/ubuntu}"
}
This syntax will work in bash as well as zsh (the double-quotes prevent unexpected word-splitting in bash, and aren't really needed in zsh).
BTW, I'm also a bit nervous about just blindly replacing "root" with "ubuntu" in the arguments; what if it occurs somewhere other than the username, like as part of a filename or hostname?
This question already has answers here:
Variables as commands in Bash scripts
(5 answers)
Closed 6 years ago.
I'm using a bash script to sync a web folder using rsync. I'm using an array of folders and files to exclude, and I'm having problems escaping these items for the exclude list...
my exclude list is defined like so...
SYNC_EXCLUSIONS=(
'/exclude_folder_1'
'/exclude_folder_2'
'.git*'
'.svn'
)
Then I build my exclusion string like so...
exclusions='';
for e in "${SYNC_EXCLUSIONS[#]}"
do
exclusions+=" --exclude='$e'";
done
Then finally I execute my rsync...
rsync --recursive --delete $exclusions "$DEPLOYMENT_WORK_DIR/" "$DEPLOYMENT_ROOT/"
If I echo the command it looks perfect, and if I copy and execute it at the prompt it works correctly. However when run from the script the exclusions are ignored.
I've figured out that it will work if I remove the single quotes from around each excluded item, like so...
exclusions+=" --exclude=$e";
I'd prefer to not do that though, just in case I need to exclude folders with spaces or special characters.
Is there some way I can get this to work from the script while retaining quotes around the excluded items? I've tried all sorts of combinations of quotes and backslashes etc. and nothing I've tried works.
You can't build a string for this at all -- see BashFAQ #50 for an extensive discussion of why. Build an array.
exclusions=( )
for e in "${SYNC_EXCLUSIONS[#]}"; do
exclusions+=( --exclude="$e" )
done
rsync --recursive --delete "${exclusions[#]}" "$DEPLOYMENT_WORK_DIR/" "$DEPLOYMENT_ROOT/"
...well, can't build a string at all, unless you're going to execute it with eval. Doing that in a manner that isn't prone to shell injection vulnerabilities takes care, however:
printf -v exclusions_str '--exclude=%q ' "${SYNC_EXCLUSIONS[#]}"
printf -v rsync_cmd 'rsync --recursive --delete %s %q %q' \
"$exclusions_str" "$DEPLOYMENT_WORK_DIR/" "$DEPLOYMENT_ROOT/"
eval "$rsync_cmd"
I'm trying to do an rsync like this in a bash script;
rsync -e "ssh ${flags_ssh}" -avRz --rsync-path="sudo rsync" $direcNew $(eval echo ${user_name})#$(eval echo ${instance_address}):$(eval echo ${mountdir}`)
However, when I run this using bash -x like this:
bash -x ./myscript
I get that it's trying to run that command except with the option rsync-path looking like
'--rsync-path=sudo rsync'
How do I prevent this? I need the double quotes to stay and the single quotes to go away, I don't know why it's happening, and I've tried endless combinations of eval and backslashes with no success.
You don't need to prevent this. What's happening is that when the shell parses --rsync-path="sudo rsync", it removes the quotes (after they have the intended effect of having the space treated as part of the argument, rather than a separator between arguments). Then, when it sees it's in -x mode, it comes up with a representation that would have led to the space being treated that way, and prints that. It could print any equivalent representation, including (but not limited to) any of these:
--rsync-path="sudo rsync"
"--rsync-path=sudo rsync"
--rsync-path=sudo\ rsync
--rsync-path=sudo" "rsync
'--rsync-path=sudo rsync'
$'--rsync-path=sudo rsync'
...etc
The fact that it picked a different representation than you did is not important, because these are all fully equivalent -- they all result in exactly the same thing being passed as an argument to rsync, so you don't need to worry about it.
You also intrinsically can't "fix" it, because by the time the shell prints its interpretation of your command, it's already forgotten which representation you happened to use -- it only knows the resulting argument that's going to be passed to rsync. In order to get it to print something else, you'd have to be passing a different actual argument to rsync, and that would break your rsync command.
This question already has answers here:
Getting "command not found" error in bash script
(6 answers)
Closed 2 years ago.
I've created a simple script to check if a folder exists and if not to create it. The script that follow
#!/bin/bash
PATH=~/Dropbox/Web_Development/
FOLDER=Test
if [ ! -d $PATH$FOLDER ]
then
echo $PATH$FOLDER 'not exists'
/bin/mkdir $PATH$FOLDER
echo $PATH$FOLDER 'has been created'
fi
works only if the mkdir command is preceded by /bin/. Failing in that, bash env output the error message "command cannot be found".
I though this could have been related to the system $PATH variable, but it looks regular (to me) and the output is as following:
/Library/Frameworks/Python.framework/Versions/2.7/bin:/bin:/usr/local/bin:/usr/bin:/sbin:/usr/local/sbin:/usr/sbin
I'm not sure whether the order with the different bin folders have been listed make any difference, but the /bin one (where the mkdir on my OSX Maverick) seems to reside is there hence I would expect bash to being able to execute this.
In fact, if I call the bash command from terminal, by typing just mkdir bash output the help string to suggest me how the mkdir command should be used. This suggests me that at a first instance bash is able to recognise the $PATH variable.
So what could be the cause? Is there any relation between the opening statement at the top of my .sh - #!/bin/bash - file and the "default" folder?
Thanks
Yeah, sometimes it is a bad idea to use capital letters for constant variables, because there are some default ones using the same convention. You can see some of the default variables here (Scroll to Special Parameters and Variables section). So it is better to use long names if you don't want to get any clashes.
Another thing to note is that you're trying to replicate mkdir -p functionality, which creates a folder if it does not exist (also it does create all of the parents, which is what you need in most cases)
One more thing - you always have to quote variables, otherwise they get expanded. This may lead to some serious problems. Imagine that
fileToRemove='*'
rm $fileToRemove
This code will remove all files in the current folder, not a file named * as you might expect.
One more thing, you should separate path from a folder with /. Like this "$MY_PATH/$MY_FOLDER". That should be done in case you forget to include / character in your path variable. It does not hurt to have two slashes, that means that /home/////////user/// folder is exactly the same /home/user/ folder.
Sometimes it is tricky to get ~ working, so using $HOME is a bit safer and more readable anyway.
So here is your modified script:
#!/bin/bash
MY_PATH="$HOME/Dropbox/Web_Development/"
MY_FOLDER='Test'
mkdir -p "$MY_PATH/$MY_FOLDER"
The problem is that your script sets PATH to a single directory, and that single directory does not contain a program called mkdir.
Do not use PATH as the name of a variable (use it to list the directories to be searched for commands).
Do learn the list of standard environment variable names and those specific to the shell you use (e.g. bash shell variables). Or use a simple heuristic: reserved names are in upper-case, so use lower-case names for variables local to a script. (Most environment variables are in upper-case — standard or not standard.)
And you can simply ensure that the directory exists by using:
mkdir -p ~/Dropbox/Web_Development
If it already exists, no harm is done. If it does not exist, it is created, and any other directories needed on the path to the directory (eg ~/Dropbox) is also created if that is missing.
I'm trying to create a bash script which will sync a directory specified as a command line parameter to a remote server (also specified by a parameter). At the moment, I'm using eval, which solves a parameter expansion problem, but for some reason causes rsync not to preserve ownership on the remote files (apart from being Evil, I know). Running the rsync command with all the same flags and parameters from the command prompt works fine.
I tried using $() as an alternative, but I got into a real mess with variable expansion and protecting the bits that need protecting for the remote rsync path (which needs both quotes and backslashes for paths with spaces).
So - I guess 2 questions - is there a reason that eval is preventing rsync from preserving ownership (the bash script is being run as root on the source machine, and sshing to the remote machine as root too - just for now)? And is there a way of getting $() to work in this scenario? The (trimmed) code is below:
#!/bin/bash
RSYNC_CMD="/usr/bin/rsync"
RSYNC_FLAGS="-az --rsh=\"/usr/bin/ssh -i \${DST_KEY}\"" # Protect ${DST_KEY} until it is assigned later
SRC=${1} # Normally this is sense checked and processed to be a canonical path
# Logic for setting DST based on command line parameter snipped for clarity - just directly assign for testing
DST='root#some.server.com:'
DST_KEY='/path/to/sshKey.rsa'
TARG=${DST}${SRC//' '/'\ '} # Escape whitespace for target system
eval ${RSYNC_CMD} ${RSYNC_FLAGS} \"${SRC}\" \"${TARG}\" # Put quotes round the paths - even though ${TARG} is already escaped
# All synced OK - but ownership not preserved despite -a flag
I've tried changing RSYNC_CMD to sudo /usr/bin/rsync, and also adding --rsync-path="sudo /usr/bin/rsync to RSYNC_FLAGS, but neither made any difference. I just can't see what I'm missing...
The correct way to do this is to use an array. -a should already imply -o.
RSYNC_CMD="/usr/bin/rsync"
DST='root#some.server.com:'
DST_KEY='/path/to/sshKey.rsa'
RSYNC_FLAGS=(-az --rsh="/usr/bin/ssh -i ${DST_KEY}")
SRC=${1}
TARG="${DST}$SRC"
${RSYNC_CMD} "${RSYNC_FLAGS[#]}" "${SRC}" "${TARG}"
Using RSYNC_RSH instead of --rsh, you can export the variable before you set its value. This at least lets you put the export in the same area where you set the rest of the flags. Then you can defer completing its value until after you have the correct identity file.
RSYNC_CMD="/usr/bin/rsync"
export RSYNC_RSH="/usr/bin/ssh -i %s" # Use a placeholder for now; set it later
RSYNC_FLAGS=( -a -z )
# Later...
DST='root#some.server.com:'
DST_KEY='/path/to/sshKey.rsa'
RSYNC_RSH=$( printf "$RSYNC_RSH" "$DST_KEY" )
SRC=${1}
TARG="${DST}$SRC"
${RSYNC_CMD} "${RSYNC_FLAGS[#]}" "${SRC}" "${TARG}"