I get tripped up on the sequence of substitutions shell does. I understand shell will do variable substitution before file substitution, which is done before the command line is parsed. Shell can do many different substitutions. I just don't know which is done first, second, third, and so forth. Does anyone have a precedence chart of shell substitutions?
From man (1) bash:
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.
Related
I have two text files identical to each other a.text and b.text with the same content.
Content of a.text and b.text
abcd
target
efgh
Can anyone explain why one of the commands work but not the other and if there is a way of making it work?
Output of command 1
grep "target" {a,b}.text
>>a.text:target
b.text:target
Output of command 2
file="{a,b}.text"
grep "target" $file
>>grep: {a,b}.text: No such file or directory
Happy if someone can point me to a location where I can read more about this as well. I can only assume that when storing it as a variable it explicitly looks for a file called {a,b}.text although, what I am not as sure about is why and what leads to that.
As user1934428 said, brace expansion happens before parameter expansion. Quoting from the bash manual:
Expansion is performed on the command line after it has been split into words. There are seven kinds of expansion performed: brace expansion, tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, word splitting, and pathname expansion.
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.
To get this to work you can store the file names in an array. Arrays can hold multiple file names without being subject to quoting or expansion issues that plague normal string variables.
files=({a,b}.text)
grep "target" "${files[#]}"
This works because {a,b} is now evaluated when the variable is assigned, rather than when it is expanded.
In the first case, you have a brace-expansion. In the second case, you are searching a file with the literal name {a,b}.text. The reason is that in bash, brace expansion happens before parameter expansion.
A common trope on StackOverflow bash is: "Why doesn't x=99; echo {1..$x} work?"
The answer is "because braces are expanded before parameters/variables".
Therefore, I thought it should be possible to expand multiple variables using a single $ and a brace. I'd expect a=1; b=2; c=3; echo ${{a..c}} to print 1 2 3. First, the inner brace would expand to ${a} ${b} ${c} (which it does when writing echo \${{a..c}}). Then that result would undergo parameter expansion.
However, I got -bash: ${{a..c}}: bad substitution so {a..c} wasn't expanded at all.
Bash's manual is a bit more specific (emphasis mine).
Expansion is performed on the command line after it has been split into tokens [...]
The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion); word splitting; and filename expansion.
Note the ; and , in that list. "Left-to-right fashion" seems to apply to the whole (therefore unordered) list before the ;. Just like the mathematical operators * and / have no precedence over each other.
Ok, so brace expansion is not really of higher precedence than parameter expansion. It's just that both {1..$x} and ${{a..c}} are evaluated from left to right, meaning the brace { comes before the parameter $x and the parameter ${ comes before the brace {a..c}.
Or so I thought. However, when using $ instead of ${ then parameters on the left expand after braces on the right:
# in bash 5.0.3(1)
x=nil; x1=one; x2=two
echo ${x{1..2}} # prints `-bash: ${x{1..2}}: bad substitution`
echo $x{1..2} # prints `one two`
Question
Could it be that the bash manual is flawed or did I read it wrong?
If the manual is flawed: What is the exact order of all expansions?
I'm just asking because I'm curious. I don't plan to use thinks like $x{1..2} anywhere. I'm not interested in better solutions or alternatives to address multiple variables (e.g. array slices ${array[#]:1:2}). I just want to get a deeper understanding.
from: https://www.gnu.org/software/bash/manual/html_node/Brace-Expansion.html
To avoid conflicts with parameter expansion, the string ‘${’ is not
considered eligible for brace expansion, and inhibits brace expansion
until the closing ‘}’.
That said, for echo $x{1..2} , first the brace expansion takes place, and then the parameter expansion, so we have echo $x1 $x2. For echo ${x{1..2}} the brace expansion doesn't happen, because we are after the ${ and haven't reached the closing } of the parameter expansion.
Regarding the bash manual part you have quoted, left-to-right order still exists for the expansions (with respect to allowed nested ones). Things get clearer if you format the list instead of using , and ;:
brace expansion
In a left-to-right fashion:
tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution
word splitting
filename expansion.
Read Mo Budlong's 1988 classic Command Line Psychology, which was written for regular Unix, but most of it still applies to bash. The order of evaluation goes:
1 History substitution (except for the Bourne shell)
2 Splitting words, including special characters
3 Updating the history list (except for the Bourne shell)
4 Interpreting single and double quotes
5 Alias substitution (except for the Bourne shell)
6 Redirection of input and output (< > and |)
7 Variable substitution (variables starting with $)
8 Command substitution (commands inside back quotes)
9 File name expansion (file name wild cards)
So what bash does with code like {1..3} happens before step 7 above, and that's why the OP code fails.
But if we must, there's always eval, (which should only be used if the variables are known in advance, or first cautiously type checked):
a=1; b=2; c=3; eval echo \{$a..$c}
Output:
1 2 3
Why is the $bar being printed here as a literal, even thought the outer subshell should expand it's parameters according to bash command line processing rules?
$ foo='$bar' bar=expanded
$ echo $(echo $(echo $foo))
$bar
The inner subshell prints $bar, but why doesn't the outer subshell expand it? Does the bash implicitly pass it as a literal and if so, why and how? According to my knowledge, the parameter expansions happens after each fork of the subshell, inside the new process. In the case of nested subshells, the command substitution is done from inside out, inner subshell printing out the literal, raw text representation of the outer shell command line before the fork happens and the command line (string of characters) is being split, expanded and processed by the new shell. Now the question is, why the text $bar is not expanded in the outer subshell, even thought it actually doesn't contain quotes? What causes it to be implicitly quoted here?
Here is example of the same logic and expected output without nested shells
$ foo='$bar' bar=expanded
$ echo $foo
$bar
$ echo $bar
expanded
Also, by adding eval I get the result which I would expect in the first example, but I don't undertand why it's necessary and how it wokrs.
$ echo $(eval echo $(echo $foo))
expanded
The Bash manual explains the ordering shell expansions: (reformatted for clarity)
The order of expansions is:
brace expansion;
tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion);
word splitting;
and filename expansion.
On systems that can support it, there is an additional expansion available: process substitution. This is performed at the same time as tilde, parameter, variable, and arithmetic expansion and command substitution.
After these expansions are performed, quote characters present in the original word are removed unless they have been quoted themselves (quote removal).
This essentially echoes the Posix shell specification with the addition of some bash-specific expansions.
Note that the second group of expansions, which includes command substitution ($(...)) is only performed once, left-to-right. They are not performed repetitively, so the result of a command substitution is not subject to parameter expansion. Unless quoted, it is subject to word-splitting, filename expansion, and quote removal.
The commands evaluated in subshells are, indeed, evaluated inside out, but at each level the inner command substitution is only subject to word-splitting, filename expansion and quote removal (none of which apply in thus example).
So the only parameter expansion done is the replacement of $foo with its value.
This question already has answers here:
Is it necessary to quote command substitutions during variable assignment in bash?
(3 answers)
Closed 6 years ago.
I always thought I must put quotes around $() in bash, like in:
FOO="$(echo "bar baz")"
but apparently this is unnecessary, at least during variable assignment:
$ FOO=$(echo "foo bar")
$ echo "$FOO"
foo bar
On the other hand, if I just try to assign multiple words to a variable, I get an error, because it's interpreted as "set variable for duration of subsequent command":
$ FOO=bar fooooo
fooooo: command not found
Also, if I just use $() without quotes in non-assignment context, they're again treated as separate words:
$ echo $(echo "baa beee")
baa beee
So, what are the rules regarding $() and "" interaction, and how safe is the non-quote variant? I'd be especially grateful for manpage quotes, or some other authoritative references. Also, is there some "good practice/style" here?
In brief, quotes are necessary to suppress otherwise normal behavior. In most contexts, you would need to quote the command substitution to suppress word-splitting and pathname expansion.
From the man page, under "Command substitution":
If the [command] substitution appears within double quotes, word
splitting and pathname expansion are not performed on the results.
However, the right-hand side of an assignment is not one of those contexts. From the man page, under "PARAMETERS":
A variable may be assigned to by a statement of the form
name=[value]
If value is not given, the variable is assigned the null string. All values
undergo tilde expansion, parameter and variable expansion, command substitution,
arithmetic expansion, and quote removal (see EXPANSION below).
Note that neither word splitting nor pathname expansion are mentioned.
As a rule, when in doubt, quote any expansion. The times when you want word-splitting and pathname expansion of an expansion are rare and usually obvious.
I have a pretty strange problem with sed, if I do with this:
[root#Camel ~]-> sed -i 's/TLRAGENT_IP=.*/TLRAGENT_IP='"${HOST_IP}"'/' ~user/.bash_profile
it's fine. But if I try the following:
[root#Camel ~]-> CONF_FILE="~user/.bash_profile"
[root#Camel ~]-> sed -i 's/TLRAGENT_IP=.*/TLRAGENT_IP='"${HOST_IP}"'/' ${CONF_FILE}
sed: can't read ~user/.bash_profile: No such file or directory
also tried to quote the variable:
[root#Camel ~]-> sed -i 's/TLRAGENT_IP=.*/TLRAGENT_IP='"${HOST_IP}"'/' "${CONF_FILE}"
sed: can't read ~user/.bash_profile: No such file or directory
couldn't figure out where went wrong, please advise.
Shell's version is 3.2.25(1)-release.
from man bash /EXPANSION
EXPANSION
Expansion is performed on the command line after it has been
split into words. There are seven kinds of expansion per-
formed: brace expansion, tilde expansion, parameter and
variable expansion, command substitution, arithmetic expan-
sion, word splitting, and pathname expansion.
The order of expansions is: brace expansion, tilde expan-
sion, parameter, variable and arithmetic expansion and com-
mand substitution (done in a left-to-right fashion), word
splitting, and pathname expansion.
The parameter and variable expansion comes after tilde expansion
To have tilde expansion, this can be done at variable definition
CONF_FILE=~user/.bash_profile
instead of
CONF_FILE="~user/.bash_profile"