I recently got this script to match a string in command line aguments:
if [[ "$#" == "${##foo}" ]]; then echo "not found" ; else echo "found"; fi
Looks like it does negative search. What is the meaning of ## exactly?
from man bash under Parameter Expansion:
${parameter#word}
${parameter##word}
Remove matching prefix pattern. The word is expanded to produce a pattern just as in pathname expansion. If the pattern
matches the beginning of the value of parameter, then the result of the expansion is the expanded value of parameter with the shortest matching pattern (the # case) or the longest matching pattern (the ## case) deleted. If parameter is # or *, the pattern removal operation is applied to
each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with # or *, the pattern
removal operation is
applied to each member of the array in turn, and the expansion is the resultant list.
The $# is expanded to the list of script arguments each enclosed into quotes:
"$1" "$2" "$3"...
thus the "${##foo}" would expand to a list of quoted script arguments with stripped out prefix foo if such prefix exist in any of expanded strings, i.e. the script test:
#!/bin/bash
echo "${##foo}"
invoked as:
./test abc foodef
outputs
abc def
Related
What the title says: what does it mean to encapsulate a variable in {}, "", or "{}"? I haven't been able to find any explanations online about this - I haven't been able to refer to them except for using the symbols, which doesn't yield anything.
Here's an example:
declare -a groups
groups+=("CN=exampleexample,OU=exampleexample,OU=exampleexample,DC=example,DC=com")
groups+=("CN=example example,OU=example example,OU=example example,DC=example,DC=com")
This:
for group in "${groups[#]}"; do
echo $group
done
Proves to be much different than this:
for group in $groups; do
echo $group
done
and this:
for group in ${groups}; do
echo $group
done
Only the first one accomplishes what I want: to iterate through each element in the array. I'm not really clear on the differences between $groups, "$groups", ${groups} and "${groups}". If anyone could explain it, I would appreciate it.
As an extra question - does anyone know the accepted way to refer to these encapsulations?
Braces ($var vs. ${var})
In most cases, $var and ${var} are the same:
var=foo
echo $var
# foo
echo ${var}
# foo
The braces are only needed to resolve ambiguity in expressions:
var=foo
echo $varbar
# Prints nothing because there is no variable 'varbar'
echo ${var}bar
# foobar
Quotes ($var vs. "$var" vs. "${var}")
When you add double quotes around a variable, you tell the shell to treat it as a single word, even if it contains whitespaces:
var="foo bar"
for i in "$var"; do # Expands to 'for i in "foo bar"; do...'
echo $i # so only runs the loop once
done
# foo bar
Contrast that behavior with the following:
var="foo bar"
for i in $var; do # Expands to 'for i in foo bar; do...'
echo $i # so runs the loop twice, once for each argument
done
# foo
# bar
As with $var vs. ${var}, the braces are only needed for disambiguation, for example:
var="foo bar"
for i in "$varbar"; do # Expands to 'for i in ""; do...' since there is no
echo $i # variable named 'varbar', so loop runs once and
done # prints nothing (actually "")
var="foo bar"
for i in "${var}bar"; do # Expands to 'for i in "foo barbar"; do...'
echo $i # so runs the loop once
done
# foo barbar
Note that "${var}bar" in the second example above could also be written "${var}"bar, in which case you don't need the braces anymore, i.e. "$var"bar. However, if you have a lot of quotes in your string these alternative forms can get hard to read (and therefore hard to maintain). This page provides a good introduction to quoting in Bash.
Arrays ($var vs. $var[#] vs. ${var[#]})
Now for your array. According to the bash manual:
Referencing an array variable without a subscript is equivalent to referencing the array with a subscript of 0.
In other words, if you don't supply an index with [], you get the first element of the array:
foo=(a b c)
echo $foo
# a
Which is exactly the same as
foo=(a b c)
echo ${foo}
# a
To get all the elements of an array, you need to use # as the index, e.g. ${foo[#]}. The braces are required with arrays because without them, the shell would expand the $foo part first, giving the first element of the array followed by a literal [#]:
foo=(a b c)
echo ${foo[#]}
# a b c
echo $foo[#]
# a[#]
This page is a good introduction to arrays in Bash.
Quotes revisited (${foo[#]} vs. "${foo[#]}")
You didn't ask about this but it's a subtle difference that's good to know about. If the elements in your array could contain whitespace, you need to use double quotes so that each element is treated as a separate "word:"
foo=("the first" "the second")
for i in "${foo[#]}"; do # Expands to 'for i in "the first" "the second"; do...'
echo $i # so the loop runs twice
done
# the first
# the second
Contrast this with the behavior without double quotes:
foo=("the first" "the second")
for i in ${foo[#]}; do # Expands to 'for i in the first the second; do...'
echo $i # so the loop runs four times!
done
# the
# first
# the
# second
TL;DR
All the examples you give are variations on Bash Shell Expansions. Expansions happen in a particular order, and some have specific use cases.
Braces as Token Delimiters
The ${var} syntax is primarily used for delimiting ambiguous tokens. For example, consider the following:
$ var1=foo; var2=bar; var12=12
$ echo $var12
12
$ echo ${var1}2
foo2
Braces in Array Expansions
The braces are required to access the elements of an array and for other special expansions. For example:
$ foo=(1 2 3)
# Returns first element only.
$ echo $foo
1
# Returns all array elements.
$ echo ${foo[*]}
1 2 3
# Returns number of elements in array.
$ echo ${#foo[*]}
3
Tokenization
Most of the rest of your questions have to do with quoting, and how the shell tokenizes input. Consider the difference in how the shell performs word splitting in the following examples:
$ var1=foo; var2=bar; count_params () { echo $#; }
# Variables are interpolated into a single string.
$ count_params "$var1 $var2"
1
# Each variable is quoted separately, created two arguments.
$ count_params "$var1" "$var2"
2
The # symbol interacts with quoting differently than *. Specifically:
$# "[e]xpands to the positional parameters, starting from one. When the expansion occurs within double quotes, each parameter expands to a separate word."
In an array, "[i]f the word is double-quoted, ${name[*]} expands to a single word with the value of each array member separated by the first character of the IFS variable, and ${name[#]} expands each element of name to a separate word."
You can see this in action as follows:
$ count_params () { echo $#; }
$ set -- foo bar baz
$ count_params "$#"
3
$ count_params "$*"
1
The use of a quoted expansion matters a great deal when variables refer to values with spaces or special characters that might prevent the shell from word-splitting the way you intend. See Quoting for more on how quoting works in Bash.
You need to distinguish between arrays and simple variables — and your example is using an array.
For plain variables:
$var and ${var} are exactly equivalent.
"$var" and "${var}" are exactly equivalent.
However, the two pairs are not 100% identical in all cases. Consider the output below:
$ var=" abc def "
$ printf "X%sX\n" $var
XabcX
XdefX
$ printf "X%sX\n" "${var}"
X abc def X
$
Without the double quotes around the variable, the internal spacing is lost and the expansion is treated as two arguments to the printf command. With the double quotes around the variable, the internal spacing is preserved and the expansion is treated as one argument to the printf command.
With arrays, the rules are both similar and different.
If groups is an array, referencing $groups or ${groups} is tantamount to referencing ${groups[0]}, the zeroth element of the array.
Referencing "${groups[#]}" is analogous to referencing "$#"; it preserves the spacing in the individual elements of the array, and returns a list of values, one value per element of the array.
Referencing ${groups[#]} without the double quotes does not preserve spacing and can introduce more values than there are elements in the array if some of the elements contain spaces.
For example:
$ groups=("abc def" " pqr xyz ")
$ printf "X%sX\n" ${groups[#]}
XabcX
XdefX
XpqrX
XxyzX
$ printf "X%sX\n" "${groups[#]}"
Xabc defX
X pqr xyz X
$ printf "X%sX\n" $groups
XabcX
XdefX
$ printf "X%sX\n" "$groups"
Xabc defX
$
Using * instead of # leads to subtly different results.
See also How to iterate over the arguments in a bash script.
The second sentence of the first paragraph under Parameter Expansion in man bash says,
The parameter name or symbol to be expanded may be enclosed in braces, which are optional but serve to protect the variable to be expanded from characters immediately following it which could be interpreted as part of the name.
Which tells you that the name is simply braces, and the main purpose is to clarify where the name begins and ends:
foo='bar'
echo "$foobar"
# nothing
echo "${foo}bar"
barbar
If you read further you discover,
The braces are required when parameter is a positional parameter with more than one digit…
Let's test:
$ set -- {0..100}
$ echo $22
12
$ echo ${22}
20
Huh. Neat. I honestly didn't know that before writing this (I've never had more than 9 positional parameters before.)
Of course, you also need braces to do the powerful parameter expansion features like
${parameter:-word}
${parameter:=word}
${parameter:?word}
… [read the section for more]
as well as array expansion.
A related case not covered above. Quoting an empty variable seems to change things for test -n. This is specifically given as an example in the info text for coreutils, but not really explained:
16.3.4 String tests
-------------------
These options test string characteristics. You may need to quote
STRING arguments for the shell. For example:
test -n "$V"
The quotes here prevent the wrong arguments from being passed to
`test' if `$V' is empty or contains special characters.
I'd love to hear the detailed explanation. My testing confirms this, and I'm now quoting my variables for all string tests, to avoid having -z and -n return the same result.
$ unset a
$ if [ -z $a ]; then echo unset; else echo set; fi
unset
$ if [ -n $a ]; then echo set; else echo unset; fi
set # highly unexpected!
$ unset a
$ if [ -z "$a" ]; then echo unset; else echo set; fi
unset
$ if [ -n "$a" ]; then echo set; else echo unset; fi
unset # much better
Well, I know that encapsulation of a variable helps you to work with something like:
${groups%example}
or syntax like that, where you want to do something with your variable before returning the value.
Now, if you see your code, all the magic is inside
${groups[#]}
the magic is in there because you can't write just: $groups[#]
You're putting your variable inside the {} because you want to use special characters [] and #. You can't name or call your variable just: # or something[] because these are reserved characters for other operations and names.
$var and ${var} are the same, if var is the name of the variable.
The braces are required when parameter is a positional parameter with more than one digit, or when parameter is followed by a character that is not to be interpreted as part of its name.
Thus, "$var" and "${var}" are the same.
However, $var and "$var" are different.
Bash will do Word Splitting for $var but not for "$var".
The shell scans the results of parameter expansion, command substitution, and arithmetic expansion that did not occur within double quotes for word splitting.
Note: Word splitting won't be performed in variable assignment:
https://www.gnu.org/software/bash/manual/html_node/Shell-Parameters.html
A variable may be assigned to by a statement of the form
name=[value]
If value is not given, the variable is assigned the null string. All values undergo tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, and quote removal (see Shell Parameter Expansion). If the variable has its integer attribute set, then value is evaluated as an arithmetic expression even if the $((…)) expansion is not used (see Arithmetic Expansion). Word splitting and filename expansion are not performed.
I want to check if an input parameter ends with ".c"? How do I check that? Here is what I got so far (Thanks for your help):
#!/bin/bash
for i in $#
do
if [$i ends with ".c"]
then
echo "YES"
fi
done
A classical case for case!
case $i in *.c) echo Yes;; esac
Yes, the syntax is arcane, but you get used to it quickly. Unlike various Bash and POSIX extensions, this is portable all the way back to the original Bourne shell.
$ [[ foo.c = *.c ]] ; echo $?
0
$ [[ foo.h = *.c ]] ; echo $?
1
for i in $#; do
if [ -z ${i##*.c} ]; then
echo "YES: $i"
fi
done
$ ./test.sh .c .c-and-more before.c-and-after foo.h foo.c barc foo.C
YES: .c
YES: foo.c
$
Explanation (thanks to jpaugh):
Iterate over command line arguments: for i in $#; do
Main trick is here: if [ -z ${i##*.c} ]; then. Here we check if length of string ${i##*.c} is zero. ${i##*.c} means: take $i value and remove substring by template "*.c". If result is empty string, then we have ".c" suffix.
Here if some additional info from man bash, section Parameter Expasion
${parameter#word}
${parameter##word}
Remove matching prefix pattern. The word is expanded to produce a pat‐
tern just as in pathname expansion. If the pattern matches the begin‐
ning of the value of parameter, then the result of the expansion is the
expanded value of parameter with the shortest matching pattern (the
``#'' case) or the longest matching pattern (the ``##'' case) deleted.
If parameter is # or *, the pattern removal operation is applied to
each positional parameter in turn, and the expansion is the resultant
list. If parameter is an array variable subscripted with # or *, the
pattern removal operation is applied to each member of the array in
turn, and the expansion is the resultant list.
The man page says that case statements use "filename expansion pattern matching".
I usually want to have short names for some parameters, so I go:
case $1 in
req|reqs|requirements) TASK="Functional Requirements";;
met|meet|meetings) TASK="Meetings with the client";;
esac
logTimeSpentIn "$TASK"
I tried patterns like req* or me{e,}t which I understand would expand correctly to match those values in the context of filename expansion, but it doesn't work.
Brace expansion doesn't work, but *, ? and [] do. If you set shopt -s extglob then you can also use extended pattern matching:
?() - zero or one occurrences of pattern
*() - zero or more occurrences of pattern
+() - one or more occurrences of pattern
#() - one occurrence of pattern
!() - anything except the pattern
Here's an example:
shopt -s extglob
for arg in apple be cd meet o mississippi
do
# call functions based on arguments
case "$arg" in
a* ) foo;; # matches anything starting with "a"
b? ) bar;; # matches any two-character string starting with "b"
c[de] ) baz;; # matches "cd" or "ce"
me?(e)t ) qux;; # matches "met" or "meet"
#(a|e|i|o|u) ) fuzz;; # matches one vowel
m+(iss)?(ippi) ) fizz;; # matches "miss" or "mississippi" or others
* ) bazinga;; # catchall, matches anything not matched above
esac
done
I don't think you can use braces.
According to the Bash manual about case in Conditional Constructs.
Each pattern undergoes tilde
expansion, parameter expansion,
command substitution, and arithmetic
expansion.
Nothing about Brace Expansion unfortunately.
So you'd have to do something like this:
case $1 in
req*)
...
;;
met*|meet*)
...
;;
*)
# You should have a default one too.
esac
if and grep -Eq
arg='abc'
if echo "$arg" | grep -Eq 'a.c|d.*'; then
echo 'first'
elif echo "$arg" | grep -Eq 'a{2,3}'; then
echo 'second'
fi
where:
-q prevents grep from producing output, it just produces the exit status
-E enables extended regular expressions
I like this because:
it is POSIX 7
it supports extended regular expressions, unlike POSIX case
the syntax is less clunky than case statements when there are few cases
One downside is that this is likely slower than case since it calls an external grep program, but I tend to consider performance last when using Bash.
case is POSIX 7
Bash appears to follow POSIX by default without shopt as mentioned by https://stackoverflow.com/a/4555979/895245
Here is the quote: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_01 section "Case Conditional Construct":
The conditional construct case shall execute the compound-list corresponding to the first one of several patterns (see Pattern Matching Notation) [...] Multiple patterns with the same compound-list shall be delimited by the '|' symbol. [...]
The format for the case construct is as follows:
case word in
[(] pattern1 ) compound-list ;;
[[(] pattern[ | pattern] ... ) compound-list ;;] ...
[[(] pattern[ | pattern] ... ) compound-list]
esac
and then http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_13 section "2.13. Pattern Matching Notation" only mentions ?, * and [].
This question already has answers here:
What is the difference between $# and $* in shell scripts?
(3 answers)
Closed 8 years ago.
It seems to me that they both store all the command-line arguments.
So is there a difference between the two?
The difference is subtle; "$*" creates one argument separated by the $IFS variable, while "$#" will expand into separate arguments. As an example, consider:
for i in "$#"; do echo "# '$i'"; done
for i in "$*"; do echo "* '$i'"; done
When run with multiple arguments:
./testvar foo bar baz 'long arg'
# 'foo'
# 'bar'
# 'baz'
# 'long arg'
* 'foo bar baz long arg'
For more details:
http://www.gnu.org/software/bash/manual/bashref.html#Special-Parameters
$*
Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, it expands to a single word with the value of each parameter separated by the first character of the IFS special variable. That is, "$*" is equivalent to "$1c$2c...", where c is the first character of the value of the IFS variable. If IFS is unset, the parameters are separated by spaces. If IFS is null, the parameters are joined without intervening separators.
$#
Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, each parameter expands to a separate word. That is, "$#" is equivalent to "$1" "$2" .... If the double-quoted expansion occurs within a word, the expansion of the first parameter is joined with the beginning part of the original word, and the expansion of the last parameter is joined with the last part of the original word. When there are no positional parameters, "$#" and $# expand to nothing (i.e., they are removed).
A key difference from my POV is that "$#" preserves the original number
of arguments. It's the only form that does.
For example, if file my_script contains:
#!/bin/bash
main()
{
echo 'MAIN sees ' $# ' args'
}
main $*
main $#
main "$*"
main "$#"
### end ###
and I run it like this:
my_script 'a b c' d e
I will get this output:
MAIN sees 5 args
MAIN sees 5 args
MAIN sees 1 args
MAIN sees 3 args
Script test.sh:
#!/bin/bash
if [[ $# -eq 0 ]]; then
echo "no arg"
else
echo "have arg"
fi
When I ran it as
./test.sh
it said "no arg", which was expected, but if I run it as
sh ./test.sh
it prints "have arg", but it you print $#, it's zero in both cases.
However, the script below
#!/bin/bash
if [ $# -eq 0 ]; then
echo "no arg"
else
echo "have arg"
fi
prints "no arg" in both cases.
Could somebody explain this? Why does [[ ... ]] interpret $# differently from [ ... ]?
The explanations I've read about [[ ... ]] weren't clear enough about this.
/bin/sh is often a different shell interpreter than bash. On my ubuntu system, it's a symlink to dash. The different shells have different syntax.
[ foo ] and test foo are equivalent in both bash and dash.
[[ foo ]] is a bash expression that is similar to [ ] tests, but with some differences that are noted in man bash.
Dash does not have a [[ command, which results in an error-exitcode, and the execution of the else branch.
[[ expression ]]
Return a status of 0 or 1 depending on the evaluation of the
conditional expression expression. Expressions are composed of
the primaries described below under CONDITIONAL EXPRESSIONS.
Word splitting and pathname expansion are not performed on the
words between the [[ and ]]; tilde expansion, parameter and
variable expansion, arithmetic expansion, command substitution,
process substitution, and quote removal are performed. Condi‐
tional operators such as -f must be unquoted to be recognized as
primaries.
When the == and != operators are used, the string to the right
of the operator is considered a pattern and matched according to
the rules described below under Pattern Matching. If the shell
option nocasematch is enabled, the match is performed without
regard to the case of alphabetic characters. The return value
is 0 if the string matches (==) or does not match (!=) the pat‐
tern, and 1 otherwise. Any part of the pattern may be quoted to
force it to be matched as a string.
An additional binary operator, =~, is available, with the same
precedence as == and !=. When it is used, the string to the
right of the operator is considered an extended regular expres‐
sion and matched accordingly (as in regex(3)). The return value
is 0 if the string matches the pattern, and 1 otherwise. If the
regular expression is syntactically incorrect, the conditional
expression's return value is 2. If the shell option nocasematch
is enabled, the match is performed without regard to the case of
alphabetic characters. Any part of the pattern may be quoted to
force it to be matched as a string. Substrings matched by
parenthesized subexpressions within the regular expression are
saved in the array variable BASH_REMATCH. The element of
BASH_REMATCH with index 0 is the portion of the string matching
the entire regular expression. The element of BASH_REMATCH with
index n is the portion of the string matching the nth parenthe‐
sized subexpression.
In my case (ubuntu 9.04) I also see the following error line ABOVE the "have arg" line:
./test.sh: 6: [[: not found
And indeed that's as it should be; /bin/sh is a symbolic link to 'dash' not 'bash'.