Why is the $bar being printed here as a literal, even thought the outer subshell should expand it's parameters according to bash command line processing rules?
$ foo='$bar' bar=expanded
$ echo $(echo $(echo $foo))
$bar
The inner subshell prints $bar, but why doesn't the outer subshell expand it? Does the bash implicitly pass it as a literal and if so, why and how? According to my knowledge, the parameter expansions happens after each fork of the subshell, inside the new process. In the case of nested subshells, the command substitution is done from inside out, inner subshell printing out the literal, raw text representation of the outer shell command line before the fork happens and the command line (string of characters) is being split, expanded and processed by the new shell. Now the question is, why the text $bar is not expanded in the outer subshell, even thought it actually doesn't contain quotes? What causes it to be implicitly quoted here?
Here is example of the same logic and expected output without nested shells
$ foo='$bar' bar=expanded
$ echo $foo
$bar
$ echo $bar
expanded
Also, by adding eval I get the result which I would expect in the first example, but I don't undertand why it's necessary and how it wokrs.
$ echo $(eval echo $(echo $foo))
expanded
The Bash manual explains the ordering shell expansions: (reformatted for clarity)
The order of expansions is:
brace expansion;
tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion);
word splitting;
and filename expansion.
On systems that can support it, there is an additional expansion available: process substitution. This is performed at the same time as tilde, parameter, variable, and arithmetic expansion and command substitution.
After these expansions are performed, quote characters present in the original word are removed unless they have been quoted themselves (quote removal).
This essentially echoes the Posix shell specification with the addition of some bash-specific expansions.
Note that the second group of expansions, which includes command substitution ($(...)) is only performed once, left-to-right. They are not performed repetitively, so the result of a command substitution is not subject to parameter expansion. Unless quoted, it is subject to word-splitting, filename expansion, and quote removal.
The commands evaluated in subshells are, indeed, evaluated inside out, but at each level the inner command substitution is only subject to word-splitting, filename expansion and quote removal (none of which apply in thus example).
So the only parameter expansion done is the replacement of $foo with its value.
Related
A common trope on StackOverflow bash is: "Why doesn't x=99; echo {1..$x} work?"
The answer is "because braces are expanded before parameters/variables".
Therefore, I thought it should be possible to expand multiple variables using a single $ and a brace. I'd expect a=1; b=2; c=3; echo ${{a..c}} to print 1 2 3. First, the inner brace would expand to ${a} ${b} ${c} (which it does when writing echo \${{a..c}}). Then that result would undergo parameter expansion.
However, I got -bash: ${{a..c}}: bad substitution so {a..c} wasn't expanded at all.
Bash's manual is a bit more specific (emphasis mine).
Expansion is performed on the command line after it has been split into tokens [...]
The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion); word splitting; and filename expansion.
Note the ; and , in that list. "Left-to-right fashion" seems to apply to the whole (therefore unordered) list before the ;. Just like the mathematical operators * and / have no precedence over each other.
Ok, so brace expansion is not really of higher precedence than parameter expansion. It's just that both {1..$x} and ${{a..c}} are evaluated from left to right, meaning the brace { comes before the parameter $x and the parameter ${ comes before the brace {a..c}.
Or so I thought. However, when using $ instead of ${ then parameters on the left expand after braces on the right:
# in bash 5.0.3(1)
x=nil; x1=one; x2=two
echo ${x{1..2}} # prints `-bash: ${x{1..2}}: bad substitution`
echo $x{1..2} # prints `one two`
Question
Could it be that the bash manual is flawed or did I read it wrong?
If the manual is flawed: What is the exact order of all expansions?
I'm just asking because I'm curious. I don't plan to use thinks like $x{1..2} anywhere. I'm not interested in better solutions or alternatives to address multiple variables (e.g. array slices ${array[#]:1:2}). I just want to get a deeper understanding.
from: https://www.gnu.org/software/bash/manual/html_node/Brace-Expansion.html
To avoid conflicts with parameter expansion, the string ‘${’ is not
considered eligible for brace expansion, and inhibits brace expansion
until the closing ‘}’.
That said, for echo $x{1..2} , first the brace expansion takes place, and then the parameter expansion, so we have echo $x1 $x2. For echo ${x{1..2}} the brace expansion doesn't happen, because we are after the ${ and haven't reached the closing } of the parameter expansion.
Regarding the bash manual part you have quoted, left-to-right order still exists for the expansions (with respect to allowed nested ones). Things get clearer if you format the list instead of using , and ;:
brace expansion
In a left-to-right fashion:
tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution
word splitting
filename expansion.
Read Mo Budlong's 1988 classic Command Line Psychology, which was written for regular Unix, but most of it still applies to bash. The order of evaluation goes:
1 History substitution (except for the Bourne shell)
2 Splitting words, including special characters
3 Updating the history list (except for the Bourne shell)
4 Interpreting single and double quotes
5 Alias substitution (except for the Bourne shell)
6 Redirection of input and output (< > and |)
7 Variable substitution (variables starting with $)
8 Command substitution (commands inside back quotes)
9 File name expansion (file name wild cards)
So what bash does with code like {1..3} happens before step 7 above, and that's why the OP code fails.
But if we must, there's always eval, (which should only be used if the variables are known in advance, or first cautiously type checked):
a=1; b=2; c=3; eval echo \{$a..$c}
Output:
1 2 3
I seem to be able to create environment variables that execute commands; like this:
$ cat ./src
FOO="echo"
$ . ./src
$ echo $FOO
echo
$ $FOO hello
hello
$
Is there a way I can modify that environment variable so that it prefixes the setting of another environment variable before the command? I.e. is there a way to work around the following problem?
$ cat ./src
FOO="MY_DIR=/tmp echo"
$ . ./src
$ echo $FOO
MY_DIR=/tmp echo
$ $FOO hello
-bash: MY_DIR=/tmp: No such file or directory
$
I.e. what I'd like to happen is to have an environment variable that does the equivalent of the following manually typed in the shell:
$ MY_DIR=/tmp echo hello
hello
$
...similar to how sans envvar-prefix, $FOO effectively had the same effect as typing echo at the shell.
/tmp/ exists of course, btw:
$ ls -ld /tmp/
drwxrwxrwt. 25 root root 500 May 19 11:35 /tmp/
$
Update:
I have a constraint that "FOO" must be invoked like $FOO hello and not FOO hello. So unfortunately a function like in #John Kugelman's (current) answer can't be a solution, even if it's more proper.
It's best to put data into variables, code into functions. Functions are more natural, expressive, and flexible than variables holding code. They look just like any other command but can take arbitrary actions, including but not limited to prepending commands and variable assignments.
foo() {
MY_DIR=/tmp echo "$#"
}
foo hello
Here "$#" is a placeholder for the arguments passed to foo().
I have a constraint that "FOO" must be invoked like $FOO hello and not FOO hello.
That constraint is impossible, I'm afraid.
I am curious about the mechanics of what's going on here: i.e. why can you make an environment variable that's sort of "aliased" to a command (I know true aliasing is something else), but that mechanism doesn't accommodate the seemingly small change to prefix "stuff" to the command?
Bash expands commands in several passes in a fixed, prescribed order. Very early on it splits the command into words and then marks the variable assignments with invisible flags. It expands $variable references in a later pass. It doesn't look at the results to see if they look like additional variable expansions. The equal signs are effectively ignored.
If you want to know the nitty gritty details, open up the Bash man page. It's incredibly long and the details are scattered throughout. Let me pull out the key sections and some choice quotes to help you digest it:
Shell Grammar, Simple Commands
A simple command is a sequence of optional variable assignments followed by blank-separated words and redirections, and terminated by a control operator.
Simple Command Expansion
When a simple command is executed, the shell performs the following expansions, assignments, and redirections, from left to right.
The words that the parser has marked as variable assignments (those preceding the command name) and redirections are saved for later processing.
The words that are not variable assignments or redirections are expanded. If any words remain after expansion, the first word is taken to be the name of the command and the remaining words are the arguments.
...
If no command name results, the variable assignments affect the current shell environment. Otherwise, the variables are added to the environment of the executed command and do not affect the current shell environment.
Expansion
Expansion is performed on the command line after it has been split into words. There are seven kinds of expansion performed: brace expansion, tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, word splitting, and pathname expansion.
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.
Expansion, Parameter Expansion
The $ character introduces parameter expansion, command substitution, or arithmetic expansion.
Assignments are marked in step 1 and variables (AKA parameters) are expanded in step 4.
The only things that happen after variable expansion are:
Word splitting. A variable can expand to multiple words if it contains whitespace. (Or to be more precise, if it contains any of the characters in the inter-field separator variable $IFS.)
Pathname expansion. Also known as globbing, or wildcards. If a variable contains *, ?, or [ they'll be expanded to the names of matching files, if there are any.
Quote removal. This pass happens after variable expansion, but it specifically does not apply to the results of any previous expansion step. So quotes the user typed are removed, but quotes that were the results of a substitution are retained.
Neither word splitting nor pathname expansion are what you need, so that's why it's not possible to store an assignment in a variable.
This question already has answers here:
Is it necessary to quote command substitutions during variable assignment in bash?
(3 answers)
Closed 6 years ago.
I always thought I must put quotes around $() in bash, like in:
FOO="$(echo "bar baz")"
but apparently this is unnecessary, at least during variable assignment:
$ FOO=$(echo "foo bar")
$ echo "$FOO"
foo bar
On the other hand, if I just try to assign multiple words to a variable, I get an error, because it's interpreted as "set variable for duration of subsequent command":
$ FOO=bar fooooo
fooooo: command not found
Also, if I just use $() without quotes in non-assignment context, they're again treated as separate words:
$ echo $(echo "baa beee")
baa beee
So, what are the rules regarding $() and "" interaction, and how safe is the non-quote variant? I'd be especially grateful for manpage quotes, or some other authoritative references. Also, is there some "good practice/style" here?
In brief, quotes are necessary to suppress otherwise normal behavior. In most contexts, you would need to quote the command substitution to suppress word-splitting and pathname expansion.
From the man page, under "Command substitution":
If the [command] substitution appears within double quotes, word
splitting and pathname expansion are not performed on the results.
However, the right-hand side of an assignment is not one of those contexts. From the man page, under "PARAMETERS":
A variable may be assigned to by a statement of the form
name=[value]
If value is not given, the variable is assigned the null string. All values
undergo tilde expansion, parameter and variable expansion, command substitution,
arithmetic expansion, and quote removal (see EXPANSION below).
Note that neither word splitting nor pathname expansion are mentioned.
As a rule, when in doubt, quote any expansion. The times when you want word-splitting and pathname expansion of an expansion are rare and usually obvious.
I get tripped up on the sequence of substitutions shell does. I understand shell will do variable substitution before file substitution, which is done before the command line is parsed. Shell can do many different substitutions. I just don't know which is done first, second, third, and so forth. Does anyone have a precedence chart of shell substitutions?
From man (1) bash:
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.
While this code works
#!/bin/bash
d="test_files/*"
for f in $d.{mp3,txt} ;do
do something
done
putting the {mp3,txt} in to a variable does not, see code below.
#!/bin/bash
a={mp3,txt}
d="test_files/*"
for f in $d."$a" ;do
do smoething
done
the output here is /*.{mp3,txt}
Putting {mp3,txt} in to an array
a=({mp3,txt})
outputs only files with the *.mp3 extension.
It doesn't work because brace expansion happens before all other expansions.
From man bash:
Brace expansion is performed before any other expansions, and any
characters special to other expansions are preserved in the result. It
is strictly textual. Bash does not apply any syntactic interpretation
to the context of the expansion or the text between the braces. To
avoid conflicts with parameter expansion, the string ‘${’ is not
considered eligible for brace expansion
You can use eval to do brace expansion stored in variables, but it is not recommended. For example:
eval echo "$d.$a"