In the dist_train.sh from mmdetection3d, what does ${#:3} do at the last line ?
I can't understand its bash grammar.
#!/usr/bin/env bash
CONFIG=$1
GPUS=$2
NNODES=${NNODES:-1}
NODE_RANK=${NODE_RANK:-0}
PORT=${PORT:-29500}
MASTER_ADDR=${MASTER_ADDR:-"127.0.0.1"}
PYTHONPATH="$(dirname $0)/..":$PYTHONPATH \
python -m torch.distributed.launch \
--nnodes=$NNODES \
--node_rank=$NODE_RANK \
--master_addr=$MASTER_ADDR \
--nproc_per_node=$GPUS \
--master_port=$PORT \
$(dirname "$0")/train.py \
$CONFIG \
--seed 0 \
--launcher pytorch ${#:3}
It is standard parameter expansion:
${parameter:offset}
${parameter:offset:length}
This is referred to as Substring Expansion. It expands to up to
length characters of the value of parameter starting at the character
specified by offset. If parameter is #, an indexed array
subscripted by # or *, or an associative array name, the
results differ as described below. If length is omitted, it expands
to the substring of the value of parameter starting at the character
specified by offset and extending to the end of the value. length
and offset are arithmetic expressions (see Shell Arithmetic).
[...]
If parameter is #, the result is length positional parameters
beginning at offset. A negative offset is taken relative to one
greater than the greatest positional parameter, so an offset of -1
evaluates to the last positional parameter. It is an expansion error
if length evaluates to a number less than zero.
The following examples illustrate substring expansion using positional parameters:
$ set -- 1 2 3 4 5 6 7 8 9 0 a b c d e f g h
$ echo ${#:7}
7 8 9 0 a b c d e f g h
$ echo ${#:7:0}
$ echo ${#:7:2}
7 8
$ echo ${#:7:-2}
bash: -2: substring expression < 0
$ echo ${#: -7:2}
b c
$ echo ${#:0}
./bash 1 2 3 4 5 6 7 8 9 0 a b c d e f g h
$ echo ${#:0:2}
./bash 1
$ echo ${#: -7:0}
Per the Bash Hackers wiki on the Positional Parameters syntax, the ${#:3} means any script argument starting at the third argument.
In other words, the ${#:3} syntax means "all arguments EXCEPT the first and second". A similar SO question exists from which you can infer the same conclusion.
A contrived example:
foo() {
echo "${#:3}"
}
foo a b c d e f g h i
# prints c d e f g h i
Great question.
In bash this is one kind of something called variable expansion. In this case the variable is $# representing all the parameters received by the program (or function), as a string.
Using the colon : means that you want to 'expand' $# to a subset of it's original string (ie. a substring).
So in this instance you're saying give me the string representing all the incoming parameters, but start from the 3rd one.
According to tldp.org, bash underscore variable is:
The underscore variable is set at shell startup and contains the absolute file name of the shell or script being executed as passed in the argument list. Subsequently, it expands to the last argument to the previous command, after expansion. It is also set to the full pathname of each command executed and placed in the environment exported to that command. When checking mail, this parameter holds the name of the mail file.
But this answer to How can I repeat a character in Bash? makes an strange use of it:
# exactly the same as perl -E 'say "=" x 100'.
echo -e ''$_{1..100}'\b='
Playing around with this variable I can't make anything out of it's semantics, so the question is what does
An string.
Followed by $_.
Followed by range expansion.
Followed by another string
mean in bash?
Sample:
$ echo $_{0..10} ; echo $_{0..10} | wc
1 0 1
$ echo ''$_{0..10}'' ; echo ''$_{0..10}'' | wc
1 0 11
$ echo ''$_{0..10}'x' ; echo ''$_{0..10}'x' | wc
x x x x x x x x x x x
1 11 22
$ echo 'x'$_{0..10}'' ; echo 'x'$_{0..10}'' | wc
x x x x x x x x x x x
1 11 22
$ echo 'ab'$_{0..10}'cd' ; echo 'ab'$_{0..10}'cd' | wc
abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd
1 11 55
echo $_{0..10}
The braces are expanded to:
echo $_0 $_1 $_2 $_3 $_4 $_5 $_6 $_7 $_8 $_9 $_10
It prints the values of eleven strangely-named variables named _0, _1, _2, and so on. They're not set—which is why you don't see anything—but if they were, you would:
$ _0=zero _1=one _2=two _3=three _4=four _5=five _6=six _7=seven _8=eight _9=nine _10=ten
$ echo $_{0..10}
zero one two three four five six seven eight nine ten
$ echo ''$_{0..10}'x'
Same thing, but now there's an x after each variable name. It's not part of the variable name. It's a separate, literal character x, as if you'd written:
echo ${_0}x ${_1}x ${_2}x ${_3}x ${_4}x ${_5}x ${_6}x ${_7}x ${_8}x ${_9}x ${_10}x
Now the output when the variables have values is:
$ _0=zero _1=one _2=two _3=three _4=four _5=five _6=six _7=seven _8=eight _9=nine _10=ten
$ echo ''$_{0..10}'x'
zerox onex twox threex fourx fivex sixx sevenx eightx ninex tenx
This should be enough to understand the other examples in your question.
It also shows that the linked answer is a poor way to repeat a string. It relies on these variables being unset. Not recommended.
That is brace expansion.
Read
info bash -n "Brace Expansion"
This expands the last command, stored in $_, with the given numbers.
This block of code is looping through a file and loading each word into a multi dimensional array.
lcv=0
declare -A db
while read line;
do
lcv1=0
echo $line
for i in $line;
do
db[$lcv,$lcv1]=$i
echo $lcv,$lcv1,${db[$lcv,$lcv1]};
#echo ${db[$lcv]}
((++lcv1))
done
((++lcv))
done < data.txt # File Contains records of 4 fields.
echo ${db[0,1]}
echo ${db[0,0]}
Little pseudo 2D array using bash
I just re-use your algorithm, whiping all echo and useless steps.
#!/bin/bash
unset x y db
y=0
declare -A db
while read line ;do
for i in $line ;do
db[$((x++)),$y]=$i
done
((y++))
x=0
done <<<$'0 1 2 3\n4 5 6 7\n8 9 a b\nc d e f'
Now if you
declare -p db x y
bash will print:
declare -A db='([0,0]="0" [0,1]="4" [0,2]="8" [0,3]="c" [3,3]="f" [3,2]="b" [3,1]="7" [3,0]="3" [2,2]="a" [2,3]="e" [2,0]="2" [2,1]="6" [1,1]="5" [1,0]="1" [1,3]="d" [1,2]="9" )'
declare -- x="0"
declare -- y="4"
At this point, I just wanna purpose to change 9th line: ((y++)) by ((y++,maxx=maxx>x?maxx:x)). This will populate maxx (to 4 in this sample)
Then inverting the array:
for i in {0..4};do # this syntax is nice, but don't support variables
for((j=0;j<y;j++)){ # this syntaxe could use variables
echo -n ${db[$i,$j]}\
}
echo
done
will print:
0 4 8 c
1 5 9 d
2 6 a e
3 7 b f
If the data.txt contains this:
$ cat data.txt
l0val0 l0val1 l0val2 l0val3
l1val0 l1val1 l1val2 l1val3
l2val0 l2val1 l2val2 l2val3
l3val0 l3val1 l3val2 l3val3
l4val0 l4val1 l4val2 l4val3
l5val0 l5val1 l5val2 l5val3
Your program produce this:
$ ./script
l0val0 l0val1 l0val2 l0val3
0,0,l0val0
0,1,l0val1
0,2,l0val2
0,3,l0val3
l1val0 l1val1 l1val2 l1val3
1,0,l1val0
1,1,l1val1
1,2,l1val2
1,3,l1val3
l2val0 l2val1 l2val2 l2val3
2,0,l2val0
2,1,l2val1
2,2,l2val2
2,3,l2val3
l3val0 l3val1 l3val2 l3val3
3,0,l3val0
3,1,l3val1
3,2,l3val2
3,3,l3val3
l4val0 l4val1 l4val2 l4val3
4,0,l4val0
4,1,l4val1
4,2,l4val2
4,3,l4val3
l5val0 l5val1 l5val2 l5val3
5,0,l5val0
5,1,l5val1
5,2,l5val2
5,3,l5val3
l0val1
l0val0
That goes to show that the value of $lcv selects each row (line), and the value of $lcv1 selects each word (record) divided on spaces or tabs.
It is working correctly from what I can see.
If we add this lines at the end of the script:
echo "end of first script"
for i in {0..5}; do
for j in {0..3}; do
printf 'db[%s,%s]=%s ' "$i" "$j" "${db[$i,$j]}"
done
echo
done
echo
declare -p db
We will get this output:
end of first script
db[0,0]=l0val0 db[0,1]=l0val1 db[0,2]=l0val2 db[0,3]=l0val3
db[1,0]=l1val0 db[1,1]=l1val1 db[1,2]=l1val2 db[1,3]=l1val3
db[2,0]=l2val0 db[2,1]=l2val1 db[2,2]=l2val2 db[2,3]=l2val3
db[3,0]=l3val0 db[3,1]=l3val1 db[3,2]=l3val2 db[3,3]=l3val3
db[4,0]=l4val0 db[4,1]=l4val1 db[4,2]=l4val2 db[4,3]=l4val3
db[5,0]=l5val0 db[5,1]=l5val1 db[5,2]=l5val2 db[5,3]=l5val3
declare -A db=([1,1]="l1val1" [1,0]="l1val0" [1,3]="l1val3" [1,2]="l1val2" [0,0]="l0val0" [0,1]="l0val1" [0,2]="l0val2" [0,3]="l0val3" [5,1]="l5val1" [5,0]="l5val0" [5,3]="l5val3" [5,2]="l5val2" [3,3]="l3val3" [3,2]="l3val2" [3,1]="l3val1" [3,0]="l3val0" [2,2]="l2val2" [2,3]="l2val3" [2,0]="l2val0" [2,1]="l2val1" [4,0]="l4val0" [4,1]="l4val1" [4,2]="l4val2" [4,3]="l4val3" )
Now, the question is: What do you think that is wrong?.
I need to do left shift and OR operations in makefile. Something like below:
a = $(b) << 2 | 0x1
…where is b is a numeric value read using a $(shell ) command in a makefile.
I tried the following but it didn't help.
a = $(shell echo $(b) << 2 | bc)
I mean a as got value of b but not the shifted value after running the script.
The version of bc I have access to doesn't support bitwise operations, so it seems to be barking up the wrong tree. The default shell gnumake uses is /bin/sh but you can make it use bash instead and then access bash's bitwise operators directly:
SHELL=bash
a := $(( $(b) << 2 | 1 ))
Thanks to #EtanReisner for pointing out that this will cause that entire string to be stored in the variable a which would then be evaluated if you use it in a recipe. If you want the computed value stored directly in a you still need to get the shell to evaluate it:
a := $(shell echo "$$(( $(b) << 2 | 1 ))" )
What I'm after is to have the most compact expression that expands the special parameter # with an offset of 2 or else to a default value of foobar if the subscript expands to the empty string or null. I tried the following notations but without luck:
"$#:2:-foobar"
"${#:2:-foobar}"
"${#:2: -foobar}"
Is there such a compact notation? Alternatively what would be a similar solution; ideally without temporary variables?
You may combine the expansion for the second parameter or its default value, followed by the expansion from the next offset.
Assuming your array is the program or function's argument array $# then,
#!/bin/bash
echo A "${#:2}"
echo B "${#:2:}" # your attempt #1
echo C "${#:2-foobar}" # your attempt #2
echo D "${#:2: -foobar}" # your attempt #3
echo E "${2:-foobar}"
echo F "$1" "${2:-foobar}" ${#:3}
G=("$1" "${2:-foobar}" ${#:3})
echo G "${G[#]}"
Will yield the desired result for line F and G (G uses a temp variable though).
Ex:
$ bash expand.sh 1
A
B
C
D
E foobar
F 1 foobar
G 1 foobar
$ bash expand.sh 1 2 3 4
A 2 3 4
B
C 2 3 4
D
E 2
F 1 2 3 4
G 1 2 3 4
If you're trying to do this with a different array than "$#", say H=(1 2 3), providing defaults to index expansions ("${H[2]:-foobar}") doesn't seem to work. Your best bet in this case, assuming you don't want to introduce temporary variables is to use a function or eval. But at that point you might be better off just adding a conditional e.g.,
# assuming that H wasn't sparse. redefine H based on its values
H=(
"${H[0]}"
$([[ -n "${H[1]}" ]] && echo "${H[1]}" || echo "foobar")
${H[#]:2}
)
But, readability will suffer.