Not critical - but I'm trying to get a deeper understanding of bash scripting and this is driving me crazy!
My goal - in a bash script:
Define a function that accepts arguments
Set a bash variable (CMD)
with the function name & arguments
Execute it by $CMD
No problem if there is no whitespace in $args - but here's a minimal script to illustrate:
#!/bin/bash
function tstArgs () {
echo "In tstArgs: ArgCnt:$#; Arg1:[$1]; Arg2:[$2]"
}
ARG1=today
ARG2=tomorrow
CMD1="tstArgs $ARG1 $ARG2"
$CMD1 #Output - As Desired: In tstArgs: ArgCnt:2; Arg1:[today]; Arg2:[tomorrow]
ARGWS1="'today with spaces'"
ARGWS2="'tomorrow with spaces'"
CMD2="tstArgs $ARGWS1 $ARGWS2"
$CMD2 #Output: In tstArgs: ArgCnt:6; Arg1:[today]; Arg2:[with]
#The dream:
ARGARR=($ARGWS1 $ARGWS2)
CMD3="tstArgs ${ARGARR[#]}"
$CMD3 #Output: In tstArgs: ArgCnt:6; Arg1:[today]; Arg2:[with]
#ETC, ETC, ETC...
This doesn't show the COUNTLESS variations I tried - single quotes, double quotes, escaping quotes, changing IFS, using parameter escape operators ${ARG1#Q}, setting args w. echo XXX - and so much more - way too many to include here, but to be clear, I didn't just jump on stackoverflow without first spending HOURS.
Weirdly, I can use params w. whitespace if I call the function directly:
tstArgs $ARG1 $ARG2
#But no variation of anything like:
CMD="tstArgs $ARG1 $ARG2"
$CMD
I'm sure it must be possible, and probably simple - but it's some permutation I just haven't been able to crack.
Of course I can work around it - but I'm stubborn & persistent & hate to give up. If anyone has any insight, I'd be very grateful, and maybe even finally get some sleep...
Don't put arguments in a string. Put them in an array. Array elements handle spaces much more gracefully:
declare -a ARGS
ARGS+=( "today with spaces" )
ARGS+=( "tomorrow with spaces" )
CMD="tstArgs"
${CMD} "${ARGS[#]}"
Alternatively:
declare -a ARGS
ARGS[0]="today with spaces"
ARGS[1]="tomorrow with spaces"
CMD="tstArgs"
${CMD} "${ARGS[#]}"
Putting quotation marks around ${ARGS[#]} on the last line makes sure that each element of the array is quoted, thus preserving the spaces.
Related
I'm writing a bash script that uses rsync to synchronize directories. According to the Google shell style guide:
Always quote strings containing variables, command substitutions, spaces or shell meta characters, unless careful unquoted expansion is required.
Use "$#" unless you have a specific reason to use $*.
I wrote the following test case scenario:
#!/bin/bash
__test1(){
echo stdbuf -i0 -o0 -e0 $#
stdbuf -i0 -o0 -e0 $#
}
__test2(){
echo stdbuf -i0 -o0 -e0 "$#"
stdbuf -i0 -o0 -e0 "$#"
}
PARAM+=" --dry-run "
PARAM+=" mirror.leaseweb.net::archlinux/"
PARAM+=" /tmp/test"
echo "test A: ok"
__test1 nice -n 19 rsync $PARAM
echo "test B: ok"
__test2 nice -n 19 rsync $PARAM
echo "test C: ok"
__test1 nice -n 19 rsync "$PARAM"
echo "test D: fails"
__test2 nice -n 19 rsync "$PARAM"
(I need stdbuf to immediately observe output in my longer script that i'm running)
So, my question is: why does test D fail with the below message?
rsync: getaddrinfo: --dry-run mirror.leaseweb.net 873: Name or service not known
The echo in every test looks the same. If I'm suppose to quote all variables, why does it fail in this specific scenario?
It fails because "$PARAM" expands as a single string, and no word splitting is performed, although it contains what should be interpreted by the command as several arguments.
One very useful technique is to use an array instead of a string. Build the array like this :
declare -a PARAM
PARAM+=(--dry-run)
PARAM+=(mirror.leaseweb.net::archlinux/)
PARAM+=(/tmp/test)
Then, use an array expansion to perform your call :
__test2 nice -n 19 rsync "${PARAM[#]}"
The "${PARAM[#]}" expansion has the same property as the "$#" expansion : it expands to a list of items (one word per item in the array/argument list), no word splitting occurs, just as if each item was quoted.
I agree with #Fred — using arrays is best. Here's a bit of explanation, and some debugging tips.
Before running the tests, I added
echo "$PARAM"
set|grep '^PARAM='
to actually show what PARAM is.** In your original test, it is:
PARAM=' --dry-run mirror.leaseweb.net::archlinux/ /tmp/test'
That is, it is a single string that contains multiple space-separated pieces.
As a rule of thumb (with exceptions!*), bash will split words unless you tell it not to. In tests A and C, the unquoted $# in __test1 gives bash an opportunity to split $PARAM. In test B, the unquoted $PARAM in the call to __test2has the same effect. Therefore,rsync` sees each space-separated item as a separate parameter in tests A-C.
In test D, the "$PARAM" passed to __test2 is not split when __test2 is called, because of the quotes. Therefore, __test2 sees only one parameter in $#. Then, inside __test2, the quoted "$#" keeps that parameter together, so it is not split at the spaces. As a result, rsync thinks the entirety of PARAM is the hostname, so fails.
If you use Fred's solution, the output from sed|grep '^PARAM=' is
PARAM=([0]="--dry-run" [1]="mirror.leaseweb.net::archlinux/" [2]="/tmp/test")
That is bash's internal notation for an array: PARAM[0] is "--dry-run", etc. You can see each word individually. echo $PARAM is not very helpful for an array, since it only outputs the first word (here, --dry-run).
Edits
* As Fred points out, one exception is that, in the assignment A=$B, B will not be expanded. That is, A=$B and A="$B" are the same.
** As ghoti points out, instead of set|grep '^PARAM=', you can use declare -p PARAM. The declare builtin with the -p switch will print out a line that you could paste back into the shell to recreate the variable. In this case, that output is:
declare -a PARAM='([0]="--dry-run" [1]="mirror.leaseweb.net::archlinux/" [2]="/tmp/test")'
This is a good option. I personally prefer the set|grep approach because declare -p gives you an extra level of quoting, but both work fine. Edit As #rici points out, use declare -p if an element of your array might include a newline.
As an example of the extra quoting, consider unset PARAM ; declare -a PARAM ; PARAM+=("Jim's") (a new array with one element). Then you get:
set|grep: PARAM=([0]="Jim's")
# just an apostrophe ^
declare -p: declare -a PARAM='([0]="Jim'\''s")'
# a bit uglier, in my opinion ^^^^
I have 2 bash scripts. One is calling another.
Caller.sh
arg1="+hcpu_extra=111 bbb"
str="-y +hcpu_extra=111 bbb"
local cmd_re="(-y)(.*)"
if [[ $str =~ $cmd_re ]]
then
opt=${BASH_REMATCH[1]}
arg=${BASH_REMATCH[2]}
echo "matched $opt"
echo "matched $arg"
fi
./callee.sh -y $arg
## ./callee.sh -y $arg1
I found if I print $arg1 and $arg, they show the same value "+hcpu_extra=111 bbb" on the screen. But when I pass them respectively to the callee.sh as the argument. I got different results . So my question is , what is the difference between $arg and $arg1 from bash interpreter's point of view? .
First, the code won't work right as posted, because local can't be used except in a function.
If you remove the local or put this in a function, the only difference between arg and arg1 is that arg starts with a space (the one that was between "-y" and "+hcpu". but since you're expanding those variables without double-quotes around them, that'll be removed... unless you changed IFS to something that doesn't contain a space.
(BTW, variable references without double-quotes and changes to IFS are both things that can have weird effects, and are best avoided when possible.)
Anyway, my summary is: the posted code doesn't show the effect you've described; you appear to have left out something important. See How to create a Minimal, Complete, and Verifiable example.
Sometimes I need to quote an entire command line for future evaluation. Usually I do that with:
printf "%q " "$#"
That's short and sweet but the output look awful. Most of the time this doesn't matter but in occasions I want to show it to the user. For example, in a history of executed commands menu that allows for re-execution of entries. That being the case, I would like to quote in a more readable form (closer to what the user itself would have done if he were in charge of quoting). So this:
search 'Wordreference (eng->spa)' utter
would be preferable to this:
search Wordreference\ \(eng-\>spa\) utter
In order to get the first quoted form I could iterate "$#" and do something like what follows for each argument:
[[ $arg == *\ * ]] && arg="'"${arg//\'/\'\\\'\'}"'"
This is not difficult at all but it involves a loop, a conditional string transformation and concatenation of the result of each iteration.
I wonder if there is a more "batteries included" command to do this kind of transformation out of the box.
In the same way you use eval to later execute the string, you can use eval to print it:
eval "echo $yourstring"
This will remove the shell escapes but keep your variable intact.
I have a script which uses the following logic:
if [ ! -z "$1" ]; then # if any parameter is supplied
ACTION= # clear $ACTION
else
ACTION=echo # otherwise, set it to 'echo'
fi
This works fine, as-is. However, in reading the Shell Parameter Expansion section of the bash manual, it seems this should be able to be done in a single step. However, I can't quite wrap my head around how to do it.
I've tried:
ACTION=${1:-echo} # ends up with $1 in $ACTION
ACTION=${1:+}
ACTION=${ACTION:-echo} # ends up always 'echo'
and a few ways of nesting them, but nesting seems to be disallowed as far as I can tell.
I realize I've already got a working solution, but now I'm genuinely curious if this is possible. It's something that would be straightforward with a ternary operator, but I don't think bash has one.
If this is possible, I'd like to see the logic to do this seeming two-step process, with no if/else constructs, but using only any combination of the Shell Parameter Expansion features.
Thank you.
EDIT for elderarthis:
The remainder of the script is just:
find . -name "*\?[NMSD]=[AD]" -exec ${ACTION} rm -f "{}" +
I just want ACTION=echo as a sanity check against myself, hence, passing any argument will actually do the deletion (by nullifying ${ACTION}, whereas passing no args leaves echo in there.
And I know TIMTOWTDI; I'm looking to see if it can be done with just the stuff in the Shell Parameter Expansion section :-)
EDIT for Mikel:
$ cat honk.sh
#!/bin/bash
ACTION=${1-echo}
echo $ACTION
$ ./honk.sh
echo
$ ./honk.sh foo
foo
The last needs to have ACTION='', and thus return a blank line/null value.
If I insisted on doing it in fewer than 4 lines and no sub-shell, then I think I'd use:
ACTION=${1:+' '}
: ${ACTION:=echo}
This cheats slightly - it creates a blank action rather than an empty action if there is an argument to the script. If there is no argument, then ACTION is empty before the second line. On the second line, if action is empty, set it to 'echo'. In the expansion, since you (correctly) do not quote $ACTION, no argument will be passed for the blank.
Tester (xx.sh):
ACTION=${1:+' '}
: ${ACTION:=echo}
echo $ACTION rm -f a b c
Tests:
$ sh xx.sh 1
rm -f a b c
$ sh xx.sh
echo rm -f a b c
$ sh xx.sh ''
echo rm -f a b c
$
If the last line is incorrect, then remove the colon from before the plus.
If a sub-shell is acceptable, then one of these two single lines works:
ACTION=$([ -z "$1" ] && echo echo)
ACTION=$([ -z "${1+X}" ] && echo echo)
The first corresponds to the first version shown above (empty first arguments are treated as absent); the second deals with empty arguments as present. You could write:
ACTION=$([ -z "${1:+X}" ] && echo echo)
to make the relation with the second clearer - except you're only going to use one or the other, not both.
Since the markdown notation in my comment confused the system (or I got it wrong but didn't get to fix it quickly enough), my last comment (slightly amended) should read:
The notation ${var:+' '} means 'if $var is set and is not empty, then use what follows the +' (which, in this case, is a single blank). The notation ${var+' '} means 'if $var is set - regardless of whether it is empty or not - then use what follows the +'. These other expansions are similar:
${var:=X} - set $var to X unless it already has a non-empty value.
${var:-X} - expands to $var if it has a non-empty value and expands to X if $var is unset or is empty
Dropping the colon removes the 'empty' part of the test.
ACTION=${1:-echo}
is correct.
Make sure it's near the top of your script before anything modifies $1 (e.g. before any set command). Also, it wouldn't work inside a function, because $1 would be the first parameter to the function.
Also check if $1 is set but null, in which case fix how you're calling it, or use ACTION=${1-echo} (note there is no :).
Update
Ah, I assumed you must have meant the opposite, because it didn't really make sense otherwise.
It still seems odd, but I guess as a mental exercise, maybe you want something like this:
#!/bin/bash
shopt -s extglob
ACTION=$1
ACTION=${ACTION:-echo}
ACTION=${ACTION/!(echo)/} # or maybe ACTION=${ACTION#!(echo)}
echo ACTION=$ACTION
It's not quite right: it gives ACTION=o, but I think something along those lines should work.
Further, if you pass echo as $1, it will stay as echo, but I don't think that's a bad thing.
It's also terribly ugly, but you knew that when asking the question. :-)
I'm going nuts with this...let me explain...
I have a very simple Bash script with a function in it that receives 3 arguments. Those arguments are all strings. Everything was working fine 'til I had to pass a string with whitespaces to the function.
I also have some test code to call the function (whose name is substracFromFile):
# try the function
PATTERN=`echo -e '<Location test14>'`
echo "PATTERN IS $PATTERN"
HEADERPATTERN=`echo -e '<Location [a-zA-Z0-9]*>'`
echo "HEADERPATTERN IS $HEADERPATTERN"
FILE="subversion.conf"
echo "FILE IS $FICHERO"
substracFromFile $PATTERN $HEADERPATTERN $FILE
The output of this is:
PATTERN IS <Location prueba14>
HEADERPATTERN IS <Location [a-zA-Z0-9]*>
FILE IS subversion.conf
PATTERN ARGUMENT IS <Location
HEADERPATTERN ARGUMENT IS prueba14>
FILE ARGUMENT IS <Location
grep: <Location: No such file or directory
expr: syntax error
As you can see, when arguments are passed to the function and I echoed them to screen they are somehow splitted in the white space...let me show you the function initial code:
function substracFromFile
{
PATTERN=$1
HEADERPATTERN=$2
FILE=$3
# Debug only
echo "HEADER ARGUMENT IS $PATTERN"
# Debug only
echo "HEADERPATTERN ARGUMENT IS $HEADERPATTERN"
# Debug only
echo "FILE ARGUMENT IS $FILE"
This was working sweet as long as passed strings had no whitespaces at all. When I call the function strings seem to be ok, but I don't understand why they are splitted once the function is called....I'm sure this is something pretty dull, but I just don't get it...
Thanks in advance,
Alex
Parts of your question are inconsistent with other parts. For example:
FILE="subversion.conf"
echo "FILE IS $FICHERO"
But I'm assuming those are just errors made while posting the question.
As ghostdog74 mentioned in a comment, you need to quote variables that contain whitespace. In particular, this line should have quotes around the variables:
substracFromFile "$PATTERN" "$HEADERPATTERN" "$FILE"
Also, I don't see why you're using echo to set variables in lines such as this one:
PATTERN=`echo -e '<Location test14>'`
This could simply be:
PATTERN='<Location test14>'
If the value sometimes has escaped special characters in it, you can use one of these methods:
PATTERN=$'value\nwith\tescapes'
which would have "value" newline "with" tab "escapes" as its value.
Also, I recommend getting in the habit of not using all-caps for variable names. This will reduce the chance of name collisions with Bash's built-in variables.