I'm trying to programmatically use rsync and all is well until you have spaces in the paths. Then I have to quote the path in the script which is also fine until it's optional.
In this case --link-dest can be optional and I've tried variations to accommodate it in both cases of it being there and not but I'm having problems when paths need to be quoted.
One option would be to use a case statement and just call rsync with two different lines but there are other rsync options that may or may not be there and the combinations could add up quickly so a better solution would be nice. Since this is my machine and no one but me has access to it I'm not concerned about security so if an eval or something is the only way that's fine.
#!/bin/bash
src='/home/x/ll ll'
d1='/home/x/rsy nc/a'
d2='/home/x/rsy nc/b'
rsync -a "$src" "$d1"
# Try 1
x1='--link-dest='
x2='/home/x/rsy nc/a'
rsync -a $x1"$x2" "$src" "$d2"
This works until you don't have a --link-dest which will look like this:
x1=''
x2=''
rsync -a $x1"$x2" "$src" "$d2"
And rsync takes the empty "" as the source so it fails.
# Try 2
x1='--link-dest="'
x2='/home/x/rsy nc/a"'
rsync -a $x1$x2 "$src" "$d2"
This fails because double quotes in x1 & x2 are taken as part of the path and so it creates a path not found error but it does of course work like this:
x1=''
x2=''
rsync -a $x1$x2 "$src" "$d2"
I've tried a couple of other variations but they all come back to the two issues above.
You can use Bash arrays to solve this kind of problem.
# inputs come from wherever
src='/home/x/ll ll'
d1='/home/x/rsy nc/a'
d2='/home/x/rsy nc/b'
x2='something'
# or
x2=''
# build the command
my_cmd=(rsync -a)
if [[ $x2 ]]; then
my_cmd+=("--link-dest=$x2")
fi
my_cmd+=("$src" "$d2")
# run the command
"${my_cmd[#]}"
First I'm constructing the command bit by bit, with quotes carefully used to preserve multi-word options as such, storing each option/argument as an element in my array, including the rsync command itself.
Then I invoke it with the weird "${my_cmd[#]}" syntax, which means expand all the elements of the array, without re-splitting multi-word elements. Note that the quotes are important: if you remove them, the multiple word values in the array get split again and you're back to square one.
The syntax "${my_cmd[*]}" also exists, but that one joins the whole array into a single word, again now what you want. You need the [#] to get each array element expanded as its own word in the result.
Google "bash arrays" for more details.
Edit: conditional pipes
If you want to pipe the output throw some other command on some condition, you can just pipe it into an if statement. This works:
"${my_cmd[#]}" |
if [[ $log_file]]; then
tee "$log_file"
fi
and tees the output to $log_file if that variable is defined and non-empty.
Related
I try to use automator for renaming multiple files
I got this far:
the first variable must be inside the Exiftool command line
in this case I selected 2 files, but that could be 1 or 100 files
how do I make this happen? is it possible to start from array key 1 instead of array key 0 for the filenames?
The standard way to do this is to store $1 in a variable, then use shift to remove it from the argument list, and then use "$#" to get all of the remaining arguments (i.e. the original "$2" "$3" "$4" ...) Something like this:
RenameTo="$1"
shift
echo "New name: $RenameTo"
echo "files:" "$#"
I'm not sure exactly what you're trying to to with exiftool, so I won't try to give that full command.
Note that the double-quotes aren't required in zsh, but they make this portable to POSIX-compliant shells. Also, echo isn't a very good way to see what a command would do, because it looses the distinction between spaces within an argument (e.g. spaces in the new name, or within a filename) and spaces between arguments (e.g. between the filenamess in a list of them).
This question already has answers here:
Why does shell ignore quoting characters in arguments passed to it through variables? [duplicate]
(3 answers)
Closed 3 years ago.
I'm trying to pass a dynamic command that executes ls as a string that lists the files of a directory that contains spaces. However, my ls command always interprets my one directory containing spaces as multiple directories no matter what I do.
Consider the following simplified version of my shell script:
#!/bin/sh
export WORK_DIR="/Users/jthoms/Dropbox (My Company)/backup-jthoms/Work"
echo "WORK_DIR=$WORK_DIR"
export LS_CMD="ls -A \"$WORK_DIR/dependencies/apache-tomcat-8.0.45/logs\""
echo "LS_CMD=$LS_CMD"
if [ -n "$($LS_CMD)" ]
then
echo "### Removing all logs"
sudo rm "$WORK_DIR/dependencies/apache-tomcat-8.0.45/logs/*"
else
echo "### Not removing all logs"
fi
This script results in the following output:
WORK_DIR=/Users/jthoms/Dropbox (My Company)/backup-jthoms/Work
LS_CMD=ls -A "/Users/jthoms/Dropbox (My Company)/backup-jthoms/Work/dependencies/apache-tomcat-8.0.45/logs"
ls: "/Users/jthoms/Dropbox: No such file or directory
ls: (My: No such file or directory
ls: Company)/backup-jthoms/Work/dependencies/apache-tomcat-8.0.45/logs": No such file or directory
### Not removing all logs
How can I correctly escape my shell variables so that the ls command interprets my directory as a single directory containing spaces instead of multiple directories?
I recently changed this script which used to work fine for directories containing no spaces but now doesn't work for this new case. I'm working on Bash on MacOSX. I have tried various forms of escaping, various Google searches and searching for similar questions here on SO but to no avail. Please help.
Variables are for data. Functions are for code.
# There's no apparent need to export this shell variable.
WORK_DIR="/Users/jthoms/Dropbox (My Company)/backup-jthoms/Work"
echo "WORK_DIR=$WORK_DIR"
ls_cmd () {
ls -A "$1"/dependencies/apache-tomcat-8.0.45/logs
}
if [ -n "$(ls_cmd "$WORK_DIR")" ]; then
then
echo "### Removing all logs"
sudo rm "$WORK_DIR/dependencies/apache-tomcat-8.0.45/logs/"*
else
echo "### Not removing all logs"
fi
However, you don't need ls for this at all (and in general, you should avoid parsing the output of ls). For example,
find "$WORK_DIR/dependencies/apache-tomcat-8.0.45/logs/" -type f -exec rm -rf {} +
You could use
# ...
if [ -n "$(eval "$LS_CMD")" ]
# ...
See http://mywiki.wooledge.org/BashFAQ/050
Or even
# ...
if [ -n "$(bash -c "$LS_CMD")" ]
# ...
But you are probably better off using a dedicated function and/or even something more specific to your problem (using find instead of ls is usually a good idea in these cases, see some examples in the answers for this question).
Use arrays, not strings, to store commands:
ls_cmd=(ls -A "$WORK_DIR/dependencies/apache-tomcat-8.0.45/logs")
echo "ls_cmd=${ls_cmd[*]}"
if [ -n "$("${ls_cmd[#]}")" ]; then …
(The syntax highlighting on the last line is incorrect and misleading: we’re not unquoting ${ls_cmd[#]}; in reality, we are using nested quotes via a subshell here.)
That way, word splitting won’t interfere with your command.
Note that you can’t export arrays. But you don’t need to, in your code.
As others have noted, it’s often a better idea to use functions here. More importantly, you shouldn’t parse the output of ls. There are always better alternatives.
I have a question concerning why this doesn't work. Probably, it's a simple answer, but I just can't seem to figure it out.
I want to move a couple of files I have. They all have the same filename (let's say file1) but they are all in different directories (lets say /tmp/dir1,dir2 and dir3). If I were to move these individually I could do something along the lines of:
mv /tmp/dir1/file1 /tmp
That works. However, I have multiple directories and they're all going to end up in the same spot....AND I don't want to overwrite. So, I tried something like this:
mv /tmp/{dir1,dir2,dir3}/file1 /tmp/file1.{a,b,c}
When I try this I get:
/tmp/file1.c is not a directory
Just to clarify...this also works:
mv /tmp/dir1/file1 /tmp/file1.c
Pretty sure this has to do with brace expansion but not certain why.
Thanks
Just do echo to understand how the shell expands:
$ echo mv /tmp/{dir1,dir2,dir3}/file1 /tmp/file1.{a,b,c}
mv /tmp/dir1/file1 /tmp/dir2/file1 /tmp/dir3/file1 /tmp/file1.a /tmp/file1.b /tmp/file1.c
Now you can see that your command is not what you want, because in a mv command, the destination (directory or file) is the last argument.
That's unfortunately now how the shell expansion works.
You'll have to probably use an associative array.
!/bin/bash
declare -A MAP=( [dir1]=a [dir2]=b [dir3]=c )
for ext in "${!MAP[#]}"; do
echo mv "/tmp/$ext/file1" "/tmp/file1.${MAP[$ext]}"
done
You get the following output when you run it:
mv /tmp/dir2/file1 /tmp/file1.b
mv /tmp/dir3/file1 /tmp/file1.c
mv /tmp/dir1/file1 /tmp/file1.a
Like with many other languages key ordering is not guaranteed.
${!MAP[#]} returns an array of all the keys, while ${MAP[#]} returns the an array of all the values.
Your syntax of /tmp/{dir1,dir2,dir3}/file1 expands to /tmp/dir1/file /tmp/dir2/file /tmp/dir3/file. This is similar to the way the * expansion works. The shell does not execute your command with each possible combination, it simply executes the command but expands your one value to as many as are required.
Perhaps instead of a/b/c you could differentiate them with the actual number of the dir they came from?
$: for d in 1 2 3
do echo mv /tmp/dir$d/file1 /tmp/file1.$d
done
mv /tmp/dir1/file1 /tmp/file1.1
mv /tmp/dir2/file1 /tmp/file1.2
mv /tmp/dir3/file1 /tmp/file1.3
When happy with it, take out the echo.
A relevant point - brace expansion is not a wildcard. It has nothing to do with what's on disk. It just creates strings.
So, if you create a bunch of files named with single letters or digits, echo ? will wildcard and list them all, but only the ones actually present. If there are files for vowels but not consonants, only the vowels will show. But -
if you say echo {foo,bar,nope} it will output foo bar nope regardless of whether or not any or all of those exist as files or directories, etc.
I'm writing a bash script that uses rsync to synchronize directories. According to the Google shell style guide:
Always quote strings containing variables, command substitutions, spaces or shell meta characters, unless careful unquoted expansion is required.
Use "$#" unless you have a specific reason to use $*.
I wrote the following test case scenario:
#!/bin/bash
__test1(){
echo stdbuf -i0 -o0 -e0 $#
stdbuf -i0 -o0 -e0 $#
}
__test2(){
echo stdbuf -i0 -o0 -e0 "$#"
stdbuf -i0 -o0 -e0 "$#"
}
PARAM+=" --dry-run "
PARAM+=" mirror.leaseweb.net::archlinux/"
PARAM+=" /tmp/test"
echo "test A: ok"
__test1 nice -n 19 rsync $PARAM
echo "test B: ok"
__test2 nice -n 19 rsync $PARAM
echo "test C: ok"
__test1 nice -n 19 rsync "$PARAM"
echo "test D: fails"
__test2 nice -n 19 rsync "$PARAM"
(I need stdbuf to immediately observe output in my longer script that i'm running)
So, my question is: why does test D fail with the below message?
rsync: getaddrinfo: --dry-run mirror.leaseweb.net 873: Name or service not known
The echo in every test looks the same. If I'm suppose to quote all variables, why does it fail in this specific scenario?
It fails because "$PARAM" expands as a single string, and no word splitting is performed, although it contains what should be interpreted by the command as several arguments.
One very useful technique is to use an array instead of a string. Build the array like this :
declare -a PARAM
PARAM+=(--dry-run)
PARAM+=(mirror.leaseweb.net::archlinux/)
PARAM+=(/tmp/test)
Then, use an array expansion to perform your call :
__test2 nice -n 19 rsync "${PARAM[#]}"
The "${PARAM[#]}" expansion has the same property as the "$#" expansion : it expands to a list of items (one word per item in the array/argument list), no word splitting occurs, just as if each item was quoted.
I agree with #Fred — using arrays is best. Here's a bit of explanation, and some debugging tips.
Before running the tests, I added
echo "$PARAM"
set|grep '^PARAM='
to actually show what PARAM is.** In your original test, it is:
PARAM=' --dry-run mirror.leaseweb.net::archlinux/ /tmp/test'
That is, it is a single string that contains multiple space-separated pieces.
As a rule of thumb (with exceptions!*), bash will split words unless you tell it not to. In tests A and C, the unquoted $# in __test1 gives bash an opportunity to split $PARAM. In test B, the unquoted $PARAM in the call to __test2has the same effect. Therefore,rsync` sees each space-separated item as a separate parameter in tests A-C.
In test D, the "$PARAM" passed to __test2 is not split when __test2 is called, because of the quotes. Therefore, __test2 sees only one parameter in $#. Then, inside __test2, the quoted "$#" keeps that parameter together, so it is not split at the spaces. As a result, rsync thinks the entirety of PARAM is the hostname, so fails.
If you use Fred's solution, the output from sed|grep '^PARAM=' is
PARAM=([0]="--dry-run" [1]="mirror.leaseweb.net::archlinux/" [2]="/tmp/test")
That is bash's internal notation for an array: PARAM[0] is "--dry-run", etc. You can see each word individually. echo $PARAM is not very helpful for an array, since it only outputs the first word (here, --dry-run).
Edits
* As Fred points out, one exception is that, in the assignment A=$B, B will not be expanded. That is, A=$B and A="$B" are the same.
** As ghoti points out, instead of set|grep '^PARAM=', you can use declare -p PARAM. The declare builtin with the -p switch will print out a line that you could paste back into the shell to recreate the variable. In this case, that output is:
declare -a PARAM='([0]="--dry-run" [1]="mirror.leaseweb.net::archlinux/" [2]="/tmp/test")'
This is a good option. I personally prefer the set|grep approach because declare -p gives you an extra level of quoting, but both work fine. Edit As #rici points out, use declare -p if an element of your array might include a newline.
As an example of the extra quoting, consider unset PARAM ; declare -a PARAM ; PARAM+=("Jim's") (a new array with one element). Then you get:
set|grep: PARAM=([0]="Jim's")
# just an apostrophe ^
declare -p: declare -a PARAM='([0]="Jim'\''s")'
# a bit uglier, in my opinion ^^^^
I'm working in a project that uses a massive amount of shellscripts for all kind of purposes, and the performance and portability are important. Some of these scripts are using configuration files, that have the following format:
VARIABLE1="value"
VARIABLE2="several words, several values"
VARIABLE3="a,list,of,words"
Then, to use these variables we just need to tdo the following:
#!/bin/sh
. /path/to/the/configuration.file
echo "Var 1 is: $VARIABLE1"
echo "Var 2 is: $VARIABLE2"
echo "Var 3 is: $VARIABLE3"
Simple, right?
Not so much. The fact is that while we can protect the scripts against modification with a simple chown root file.sh; chmod 0711 file.sh, the configuration files must be writable, and then we find out that nasty things like this can happen:
VARIABLE1="value"; rm requiredfile.data
VARIABLE2="you dont want to see this: `rm anotherimportantfile.data`
rm thelastrequiredfile.bin
So when the configuration file is invoked, the instructions inserted in it will be executed with the privileges of any user that actually invokes it.
I know what I'm asking is tricky, but I would like to be able to filter all dangerous syntax that can lead to unauthorized code execution.
What I've done so far:
FILTER='
/^$/d # Delete empty lines
/^#/d # Delete comments
/^[A-Z0-9_]\+=.\+$/{ # Select assignments
/`/p # alert with `
/\$/p # alert with $
/\\/p # alert with \
/;/p # alert with ;
d # Accept the rest
}
'
C=`sed -e "$FILTER" $1 | wc -l` 2>/dev/null
if test $C -gt 0; then
echo "#ERR Suspicious strings in configuration file"
fi
What am I missing? Any improvements?
PS: I know that it could be possible to read safely each variable with a grep/cut combination, but it's out of the question for performance issues.
An often cited security paradigm is that you should enumerate the permitted patterns, not try to enumerate all the possible disallowed patterns.
If you restrict yourself to variables being assigned values that are always single-quoted strings, the only pattern you have to allow is
^[A-Za-z_][A-Za-z0-9_]*='[^']*'[\t ]*$
The trailing whitespace isn't strictly necessary (and if you want to be nice, you could allow for leading whitespace as well).
Single quotes inhibit all shell metacharacters; any string in single quotes is taken verbatim.
Allowing double quotes or unquoted strings is simply inviting trouble. The transition to single quotes might be a bit of a hassle, but if you came here for security advice, that's what you'll get.
Incidentally, you can simply use grep to look for violations:
if grep -v "^[A-Za-z_][A-Za-z0-9_]*='[^']*'[\t ]*$" configfile /dev/null >&2; then
echo "$0: Invalid lines in configfile -- aborting" >&2
exit 2
fi
. configfile
:
:
As a user, I would appreciate the diagnostic message to include the violations. This also avoids the cumbersome detour via wc -l.
forget the filter. My understand for this question is, you shouldn't give the write/read permission on config file directly to someone. you should only assign sudo permission to one group, add related users into that group.