In ABS guide, I could see below snippet
var1=5
var2=4
if (( var1 > var2 ))
then
echo "$var1 is greater than $var2"
fi
I am not able to understand, why we don't need $ symbol. I added $ symbol, shellcheck shows "$ symbol is not necessary on arithmetic variables".
I am still not able to understand how that dereferencing of var1 and var2 works...
Expressions inside ((...)) are evaluated in arithmetic context. Strings which can be variable names are considered as variables whose values are integers, since evaluating these strings as literal strings makes no sense in arithmetic context. These considerations are also valid for C style for loops: in for ((i = 0; i < 10; ++i)), preceding i with $ is not necessary (but it may be necessary, depending on the context, within the body of the loop).
(( ... )) follows the same evaluation rules as defined in the POSIX specification for arithmetic expressions. (The main difference is ((...)) produces an exit status reflecting if the result is zero/nonzero, while $((...)) produces the result as a string.) In particular:
If the shell variable x contains a value that forms a valid integer constant, optionally including a leading <plus-sign> or <hyphen-minus>, then the arithmetic expansions "$((x))" and "$(($x))" shall return the same value.
The shell variable var1 contains an integer constant, so $((var1)) and $(($var1)) are equivalent. This holds recursively as well.
Various shells seem to treat variables that refer to variables differently. Nothing in the POSIX wording seems to require the following kind of recursive evaluation, though both bash and dash do:
$ foo=bar
$ bar=5
$ echo $((foo)) # foo/$foo evaluates to bar, which contains an integer constant
5
bash seems to take it a step further, allowing any string to expand, followed by an attempt to evaluate the result as an arithmetic expression.
$ foo="x + 3"
$ x=5
$ echo $((foo)) # foo evaluates to x + 3, which evaluates to 5 + 3
8
but in dash:
$ foo="x+3"
$ x=5
$ echo $((foo))
dash: 3: Illegal number: x+3
It's documented in the manual in 6.5 Shell Arithmetic. It says
Within an expression, shell variables may also be referenced by name without using the parameter expansion syntax.
This features makes arithmetic expressions so much easier to read.
Interestingly, you can also refer to array elements like this as well
values=(42 54) i=0 j=1
echo $(( values[i] + values[j] )) # => 96
Other places you don't need $ (i.e. other arithmetic "contexts"):
For numerically-indexed arrays, the index (${values[i]} not ${values[$i]})
the offset and length parts of the ${var:offset:length} parameter expansion.
You don't need the $ if it's inside the (( )) for doing arithmetic.
I am still not able to understand how that dereferencing of var1 and var2 works
bash parses things differently inside the (( )), undoubtedly because it makes it much easier to read complex arithmetic expressions without the $.
Related
This question already has answers here:
How do I iterate over a range of numbers defined by variables in Bash?
(20 answers)
Variables in bash seq replacement ({1..10}) [duplicate]
(7 answers)
Closed 5 years ago.
#!/bin/sh
for i in {1..5}
do
echo "Welcome"
done
Would work, displays Welcome 5 times.
#!/bin/sh
howmany=`grep -c $1 /root/file`
for i in {1..$howmany}
do
echo "Welcome"
done
Doesn't work! howmany would equal 5 as that is what the output of grep -c would display. $1 is parameter 1 which is specific when running the script.
Any ideas?
Workarounds for not being able to use variables in a sequence brace expression:
If the intent is merely to iterate over numbers in a range - as in the OP's case - the best choice is not to use brace expansion, but instead use bash's C-style loop - see user000001's answer.
If the specific numbers aren't important and you simply need to execute a loop body a specified number of times, Cole Tierney's answer is an option.
If use of brace expansion is desired nonetheless:
If you do NOT need the numbers in the list to have a prefix or postfix, use the seq utility with an unquoted command substitution (small caveat: seq is NOT a POSIX utility, but it is widely available); e.g.
echo $(seq 3) -> 1 2 3; start number 1 implied
echo $(seq -f '%02.f' 3) -> 01 02 03 - zero-padded
echo $(seq 2 4) -> 2 3 4; explicit start and end numbers
echo $(seq 1 2 5) -> 1 3 5; custom increment (the 2 in the middle)
If you DO need the numbers in the list to have a prefix or postfix, you have several choices:
Use the seq utility with its -f option for providing a printf-style format string (as used above for zero-padding), or pure Bash workarounds based on eval (extra care needed!) or building an array in a loop, all of which are detailed in this answer.
You could also consider implementing the functionality generically, such as by writing a custom shell function or a custom script with utilities such as awk or perl.
Example of safe use of eval with variables driving a sequence brace expression:
The variables are validated beforehand, to make sure they contain decimal integers.
from=1 to=3 # sample values
# Ensure that $from and $to are decimal numbers and abort, if they are not.
(( 10#$from + 10#$to || 1 )) 2>/dev/null || { echo "Need decimal integers" >&2; exit 1; }
eval echo "A{$from..$to}" # -> 'A1 A2 A3'
General overview of brace expansion
The main purpose of brace expansion is to expand to a list of tokens with each token having an optional prefix and/or postfix; brace expansions must be unquoted and come in 2 flavors:
a fixed series (list) of comma-separated strings - variables supported
specifies and expands to a fixed number of tokens (2 or more); e.g.:
echo A{b,c,d} -> Ab Ac Ad, i.e., 3 tokens, as implied by the number of args.
echo {/,$HOME/}Library e.g., -> /Library /User/jdoe/Library
Variable references - and even globs - are supported, but note that they get expanded after brace expansion, in its result, in the course of normal evaluation.
a sequence expression (range) with .., typically numerical - variables NOT supported
expands to a variable number of tokens, driven by literal start and end points (for historical reasons, use of variables is NOT supported - see the comments on user000001's answer):
[rare] strings: only single English letters allowed; e.g. {a..c}
numbers: decimal integers only; e.g., {1..10}, {10..1}, {-1..2}
example with prefix and postfix: A{1..3}# -> A1# A2# A3#
broken example with variables: {$from..$to} # !! FAILS - $from and $to are interpreted as literals and therefore not recognized as either a single letter or a decimal integer - no brace expansion is performed (see below).
by contrast, using variables does work in zsh and ksh.
bash 4+ adds two features:
optional increment step value:
echo A{1..5..2} -> A1 A3 A5 - numbers incremented by 2
ability to zero-pad:
echo A{001..003} -> A001 A002 A003
An invalid brace expression is not expanded (treated like a regular unquoted string, with { and } treated as literals):
echo {} -> '{}' - invalid as a brace expr.: at least 2 ,-separated tokens needed
This allows the use of unquoted {} with find, for instance.
echo {1..$to} -> '{1..<value-of-$to>}' - invalid as a brace expr. in bash: variables not supported; however, valid in ksh and zsh.
(fish, by contrast, expands any {...} sequence; similarly, zsh has option BRACE_CCL (OFF by default) for expanding individual characters inside {..}, which effectively causes expansion of any nonempty {...} sequence.)
The brace expansion is evaluated before the variables are expanded. You need a c-style for loop instead:
for ((i=1;i<=howmany;i++))
do
echo "Welcome"
done
create a sequence to control your loop
for i in $(seq 1 $howmany); do
echo "Welcome";
done
The problem is that the "brace expansion" is performed before the "variable expansion"
for i in $(seq 1 $howmany)
works as #damienfrancois said, or, if you would like:
for i in $(eval echo "{$start..10}")
probably does, but don't use it for everyone's sanity.
You could also use a while loop:
while ((howmany--)); do
echo "Welcome"
done
We could also use eval in this case:
howmany=`grep -c $1 /root/file`
for i in $(eval echo {1..$howmany}); do
echo "Welcome"
done
All of these options below work for setting i to count.
count=5
for (( i=count; i>=0; i-- )); do
echo "$i"
done
for (( i=$count; i>=0; i-- )); do
echo "$i"
done
for (( i=$((count)); i>=0; i-- )); do
echo "$i"
done
for (( i=${count}; i>=0; i-- )); do
echo "$i"
done
Which is correct or preferred?
Generally, in arithmetic contexts such as ((...)) and $((...)), reference variables by name only, without the $ prefix (as your commands are already doing with respect to variable $i):
for (( i=count; i>=0; i-- )); do
echo "$i"
done
Since ((...)) is an arithmetic context itself, there is no good reason to use a separate, expanding arithmetic context - $((count)) - inside of it.
Note that $count and ${count} are equivalent, and enclosing the variable name - count - in { and } after $ is only necessary to disambiguate the variable name from subsequent characters that can also legally be part of a variable name (which doesn't apply to your commands).
As Gordon Davisson points out, some people choose to always use the ${var} form for visual clarity.
While $-prefixed variable references in arithmetic contexts do work, there is rarely a good reason to use them: the use of $ introduces an extra expansion step before the arithmetic evaluation, which is not only unnecessary, but can result in different behavior, as explained in rici's helpful answer.
The only cases where you need the $ prefix:
To reference positional and special parameters (variables) that can only ever be referenced with $: Thanks, rici.
Positional parameters: $1, $2, ...
The count of positional parameters: $#
Special parameters: $?, $$, $! (there are others, but they are not generally numeric - see chapter Special Parameters in man bash).
If you need a nonzero default value that you provide via a parameter expansion; e.g., ${count:-2} defaults to 2 if $count is unset or empty.
If you want to use a variable value as an operator rather than operand; e.g.:
op='*'; echo $(( 2 $op 2 )) - this wouldn't work with just op.
There is a difference between using $x and just x in an arithmetic context:
x causes the value of x to be evaluated as a number. If x hasn't been defined, the result is 0. If the value of x is a valid arithmetic expression, that expression is evaluated. If it is an invalid expression, the result is a syntax error.
$x causes the value of x as a string to be interpolated into the arithmetic expression, which will then be evaluated.
This leads to different evaluations, particularly with uninitialized variables:
$ unset x
$ echo $((x/2))
0
$ echo $(($x/2))
bash: /2: syntax error: operand expected (error token is "/2")
# Also, with incomplete expressions
$ x=42+
$ echo $((x 7))
bash: 42+: syntax error: operand expected (error token is "+")
$ echo $(($x 7))
49
In both cases, I prefer the behaviour associated with the unadorned use of the variable name. Consequently, I recommend its consistent use in arithmetic expressions unless you have a really good reason not to (in which case, you probably should quote the expansion to make it clearer what your expectation is.)
I would avoid $((count)), as this creates an unnecessary arithmetic evaluation step. The other ones have different syntax, but the exact same semantics. I would use the first one, for no better reason than it being shorter.
I suggest the still equivalent (in its result), but shorter :
for (( i=count+1; i-->1 ;)); do
echo "$i"
done
I have always liked that one, because it visually says "i goes toward 0".
I'd like to have some explanation on how to best use variables in bash conditional expressions [[...]].
I usually write if statement in this way:
var=1;
# some code...
if [[ $var -eq 1 ]]; then echo "ok"; else echo "fail"; fi
and this return ok as I expected.
Now I saw in some script the same similar statement like:
var=1;
# some code...
if [[ var -eq 1 ]]; then echo "ok"; else echo "fail"; fi
The only difference is the missing parameter expansion character $ in the conditional expression [[...]].
I actually expected this statement to give an error, but this syntax is accepted and returns the ok string.
I tested this statement using bash (GNU bash, version 4.3.46), zsh (5.1.1), ksh (93u+ 2012-08-01) and busybox ash (BusyBox v1.23.2).
I only get an error with busybox shell:
ash: var: bad number
I saw in the bash man page, in the ARITHMETIC EVALUATION paragraph, that:
Within an expression, shell variables may also be referenced by name without using the parameter expansion syntax
But I didn't find anything special related to parameter expansion in the CONDITIONAL EXPRESSIONS paragraph.
So, should conditional expression contain $ when referring to variable or not? and why?
The trigger here is -eq; since it is defined to perform integer comparison, its operands are evaluated in an arithmetic context. This isn't explicitly documented, though.
You should use the $, though. [[ is an extension, so there is no guarantee that it will behave identically in every shell that defines such a construct. In fact, I wouldn't even assume that [[ var -eq 3 ]] will continue to behave this way in future versions of the same shell. (( var == 3 )) is, though, documented to perform expansion of var since you are in a explicit arithmetic context.
Check the bash man page's sections on Compound Commands. In particular, the following:
((expression))
The expression is evaluated according to the rules described below
under ARITHMETIC EVALUATION. If the value of the expression is non-zero,
the return status is 0; otherwise the return status is 1. This is exactly
equivalent to `let "expression"`.
[[ expression ]]
Return a status of 0 or 1 depending on the evaluation of the conditional
expression expression. Expressions are composed of the primaries
described below under CONDITIONAL EXPRESSIONS.
If you are evaluating things that require arithmetic, use arithmetic evaluation, and check the CONDITIONAL EXPRESSIONS section for the various things you can do with [[ ... ]]. Conditions in double-square-brackets can evaluate both strings and integers, and sometimes those work the same way .. sometimes not.
From the bash man page, under CONDITIONAL EXPRESSIONS:
string1 == string2
string1 = string2
True if the strings are equal. = should be used with the test command for POSIX conformance. When used with the [[ command, this performs pattern
matching as described above (Compound Commands).
...
arg1 OP arg2
OP is one of -eq, -ne, -lt, -le, -gt, or -ge. These arithmetic binary operators return true if arg1 is equal to, not equal to, less than, less than
or equal to, greater than, or greater than or equal to arg2, respectively. Arg1 and arg2 may be positive or negative integers.
Obviously, the string "1" is the string "1", so if n=1, and you compare $n against the string "1", you'll get a hit. But you should know what you're doing, and that this is not a numeric comparison. And similarly, < and > are NOT numeric comparisons, they are string comparisons. So while "1" < "2", you may be surprised that "11" < "2" as well.
That said, bash is forgiving about what kind of conditions you ask it to evaluate:
bash-4.4$ n=1
bash-4.4$ [[ n -eq 1 ]] && echo yes
yes
bash-4.4$ [[ $n -eq 1 ]] && echo yes
yes
bash-4.4$ (( n == 1 )) && echo yes
yes
bash-4.4$ (( n = 2 )) && echo yes
yes
bash-4.4$ echo "$n"
2
The first one works because n can't be anything but a variable in this context, so bash treats it as such. But you shouldn't rely on this behaviour. Use dollar signs for variables in conditional expressions, and stick with the bash mantra, "always quote your variables".
Inside a double-square-bracket expression in bash, you should use the arithmetic binary operators if you intend your comparison to be of integers.
Note that your busybox build appears to be using ash, which is NOT bash. Ash is the "Almquist shell", an older POSIX shell than bash, written in the late 1980s. Ash is the basis for /bin/sh in FreeBSD, in addition to being preferred often over bash in embedded systems (hence busybox) due to its smaller size.
You might want to consider writing POSIX shell scripts instead of Bash shell scripts. This will mean simplifying some things, as well as jumping through some hoops for others. POSIX does not include double-square-bracket expressions, but it does allow things like [ "$n" -eq 1 ] or [ $(( n + 1 )) -eq 2 ]. And it'll work in busybox. :)
I have a script named build_other:
count=0
while [ $count -lt 6 ]
do
./build
count+=1
done
From this script, you can see I am calling a second script, named build:
echo "building job"
sleep 30s
echo "wake after sleep"
echo "file build" >> output.txt
I expect that build should be triggered 5 times, because count increases by 1 after build runs. This means output.txt should have 5 lines saying file build. The following is the output I actually received.
building job
wake after sleep
building job
wake after sleep
The file output.txt has only two lines in it.
file build
file build
Why is the loop not running 5 times as I expected?
The += operator does not do what you think here; it is not an "add to" operator, it is acting as a string concatenation operator.
$ ./foo.sh
./build
count is now 01
./build
count is now 011
After two iterations, count is 011. Using the numeric less-than operator -lt, the string 011 is converted to the numeric 11. Since 11 is not less than 6, the loop ends.
Assuming you are using bash or another modern Bourne-family shell, you can solve this in a few ways.
# Using arithmetic expansion -- $(( expression ))
count=$((count + 1))
# Increment variable in an arithmetic evaluation context -- (( ))
(( count++ ))
# When declaring count, declare that it is an integer value. Declared an
# integer, the += operator will behave as you originally expected it would.
declare -i count=0
A few excerpts from the bash man page are below.
On the behavior of the += operator:
In the context where an assignment statement is assigning a value to a shell variable or array index, the += operator can be used to append to or add to the variable's previous value. When += is applied to a variable for which the integer attribute has been set, value is evaluated as an arithmetic expression and added to the variable's current value, which is also evaluated. When += is applied to an array variable using compound assignment (see Arrays below), the variable's value is not unset (as it is when using =), and new values are appended to the array beginning at one greater than the array's maximum index (for indexed arrays) or added as additional key-value pairs in an associative array. When applied to a string-valued variable, value is expanded and appended to the variable's value.
On arithmetic evaluation:
Arithmetic expansion allows the evaluation of an arithmetic expression and the substitution of the result. The format for arithmetic expansion is:
$((expression))
The old format $[expression] is deprecated and will be removed in upcoming versions of bash.
The expression is treated as if it were within double quotes, but a double quote inside the parentheses is not treated specially. All tokens in the expression undergo parameter and variable expansion, command substitution, and quote removal. The result is treated as the arithmetic expression to be evaluated. Arithmetic expansions may be nested.
On arithmetic evaluation context:
((expression))
The expression is evaluated according to the rules described below under ARITHMETIC EVALUATION. If the value of the expression is non-zero, the return status is 0; otherwise the return status is 1. This is exactly equivalent to let "expression".
I had used several ways to do some simple integer arithmetic in BASH (3.2). But I can't figure out the best (preferred) way to do it.
result=`expr 1 + 2`
result=$(( 1 + 2 ))
let "result = 1 + 2"
What are the fundamental differences between those expressions?
Is there other ways to do the same?
Is the use of a tool like bc mandatory for floating point arithmetic?
result=`echo "7/354" | bc`
In Bash, let allows multiple assignments on a line:
let a=3 b=4 c=5
As you show in your question, quoting the argument to let allows you to put spaces around the operators. You can, however, omit the quotes if you avoid using spaces.
Another form using double parentheses at the beginning of the statement (instead of the i=$((j + 1)) form) allows you to include spaces around the equal sign or do post- or pre- increment or decrement and additional assignment operations:
(( a = ( b + c ) * 4 ))
(( count++ ))
(( d = --c**2 ))
(( e *= 2 ))
(( f = 3, g = 5 )) # multiple operations require a comma separator
If you do help "((" it says that the double parentheses is 'Equivalent to "let EXPRESSION".'
You can use the declare builtin to make assignments, including indirectly:
blue=2
name=blue
declare $name=4
echo $blue # result: 4
echo ${!name} # result: 4
Edit:
The $(()) construct is called "arithmetic expansion" and causes the contents to be evaluated as an integer expression. It's a syntax element of the shell.
If a variable is declared as an integer you don't need to use either form of double parentheses, you can omit the dollar sign from the variable name (as in the double-parentheses forms), but you can't add spaces around operators:
declare -i x=1 # set integer, initialize to 1
declare +i s=1 # clear integer, initialize to 1
x+=1 # could also be x=x+1
echo $x # result: 2 (addition)
s+=1 # could also be s=$s+1, requires a "$"
echo $s # result: 11 (string concatenation)
Unlike the forms above, calling expr involves spawning an external executable which can be quite expensive for a lot of calculations in a loop. The only time it should be used is in environments where the shell can't do its own arithmetic or for portability when a script may find its way into such an environment. POSIX shells have arithmetic capability so it would be a concern only with older systems.
Regarding the use of bc for floating point arithmetic, it or something similar is required when using Bash and many other shells. POSIX says that "Only signed long integer arithmetic is required."
Two shells that do support float math are ksh and zsh. In addition to bc, you can use dc, AWK, Python, Perl and others from within a Bash script.
One thing that Bash will do with floating point numbers is print them with the printf builtin (note that there is also an external printf, but builtins have priority).
printf "%'14.4f\n" 1234.56 # result " 1,234.5600" (in my locale)
I prefer your second option, since it doesn't need an external utility:
result=$(( 1 + 2 ))
The first option calls out to expr to do the math - I'm not familiar with let. Another alternative to bc is dc. Choose your favourite.
I can't say it's "mandatory" but bc is probably your best bet for general purpose arithmetic.
For something fancier, you can always pipe through Perl.
The downside of both thee approaches is that they both open a child process, so doing it in a tight loop will be slower than native bash expressions (same problem arises with use of backticks, in your first example). I'm not sure whether $(()) calls a child process.
Is the use of a tool like bc mandatory for floating point arithmetic?
No, if you are using a shell that supports floating point, eg zsh, ksh. Otherwise, if you want to do more advanced floating point maths, use either one of these, bc/awk/dc. Of course, Perl/Python etc as well.
The third option you have is a lot less readable as it does not look like an assignment operator. The first, as noted by others, calls an external command