I'll start with the two motivating examples, to give proper context for the question, and then ask the question. First consider this example:
$ ext=.mp3
$ fname=file.mp3
$ echo ${fname%"$ext"}
file
Evidently, in parsing ${fname%"$ext"}, bash first expands $ext into .mp3, and then expands ${fname%.mp3} into file — the last step follows trivially from the definition of % expansions. What's confusing me is the expansion of $ext...
In particular, let's compare the above with this similar example:
$ a=value
$ b=a
$ echo ${$b}
-bash: ${$b}: bad substitution
Of course, I know I could use "indirect expansion" here to achieve what I want:
$ echo ${!b}
value
But that's not relevant to my question. I want to understand the specific bash evaluation and parsing rules that explain why ${$b} fails but ${fname%"$ext"} succeeds.
The only relevant passage I've found in man bash is this:
The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion); word splitting; and pathname expansion.
But I'm not seeing how the different behaviors result from these rules.
I'd like see an explanation that explains each step of the evaluation process of the two examples, and the rule underlying each step.
If you look up ${parameter%word} expansion in the bash manual you'll see that parameter and word are treated differently. word is subject to pathname expansion while parameter is not.
${parameter%word}
${parameter%%word}
Remove matching suffix pattern. The word is expanded to produce a
pattern just as in pathname expansion. If the pattern matches a
trailing portion of the expanded value of parameter, then the result
of the expansion is the expanded value of parameter with the shortest
matching pattern (the % case) or the longest matching pattern (the
%% case) deleted. If parameter is # or *, the pattern removal
operation is applied to each positional parameter in turn, and the
expansion is the resultant list. If parameter is an array variable
subscripted with # or *, the pattern removal operation is applied to
each member of the array in turn, and the expansion is the resultant
list.
That seems like it would explain it. But it doesn't. Pathname expansion only means globbing and pattern matching with *, ?, and the like. It doesn't include variable expansion.
The key is to read up. There's a preamble that applies to the above:
In each of the cases below, word is subject to tilde expansion, parameter expansion, command substitution, and arithmetic expansion.
In totality, word is subject to all of these expansions. Key to this question: $ext is expanded via recursive parameter expansion.
I say "recursive" because it can in fact be nested arbitrarily deep. To wit:
$ echo ${fname%.mp3}
file
$ echo ${fname%"$ext"}
file
$ echo ${fname%"${ext%"$empty"}"}
file
$ echo ${fname%"${ext%"${empty%""}"}"}
file
Related
This question already has an answer here:
What is the meaning of `//` in Bash parameter expansions?
(1 answer)
Closed last year.
I am currently moving our shell/bash scripts from a jenkinsfile to groovy scripts that are stored in methods but still execute as sh scripts.
The issue i have is with variables containing // /_
exmaple:
${VARIABLE_NAME// /_}
I cannot find what // /_ exactly does when supplied like this in a variable.
I need to find another way to do this because when moved to Groovy methods, it causes formating issues where escaping doesnt work properly.
It will replace all spaces with underscores.
Consider the following example:
$ var='hello world john doe'
$ echo "${var// /_}"
hello_world_john_doe
$
${parameter/pattern/string}
The pattern is expanded to produce a pattern just as in filename expansion. Parameter is expanded and the longest match of pattern against its value is replaced with string. The match is performed according to the rules described below (see Pattern Matching). If pattern begins with ‘/’, all matches of pattern are replaced with string. Normally only the first match is replaced. If pattern begins with ‘#’, it must match at the beginning of the expanded value of parameter. If pattern begins with ‘%’, it must match at the end of the expanded value of parameter. If string is null, matches of pattern are deleted and the / following pattern may be omitted. If the nocasematch shell option (see the description of shopt in The Shopt Builtin) is enabled, the match is performed without regard to the case of alphabetic characters. If parameter is ‘#’ or ‘’, the substitution operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with ‘#’ or ‘’, the substitution operation is applied to each member of the array in turn, and the expansion is the resultant list.
Copied from: https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html with my emphasize on how / and // differences.
I have two text files identical to each other a.text and b.text with the same content.
Content of a.text and b.text
abcd
target
efgh
Can anyone explain why one of the commands work but not the other and if there is a way of making it work?
Output of command 1
grep "target" {a,b}.text
>>a.text:target
b.text:target
Output of command 2
file="{a,b}.text"
grep "target" $file
>>grep: {a,b}.text: No such file or directory
Happy if someone can point me to a location where I can read more about this as well. I can only assume that when storing it as a variable it explicitly looks for a file called {a,b}.text although, what I am not as sure about is why and what leads to that.
As user1934428 said, brace expansion happens before parameter expansion. Quoting from the bash manual:
Expansion is performed on the command line after it has been split into words. There are seven kinds of expansion performed: brace expansion, tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, word splitting, and pathname expansion.
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.
To get this to work you can store the file names in an array. Arrays can hold multiple file names without being subject to quoting or expansion issues that plague normal string variables.
files=({a,b}.text)
grep "target" "${files[#]}"
This works because {a,b} is now evaluated when the variable is assigned, rather than when it is expanded.
In the first case, you have a brace-expansion. In the second case, you are searching a file with the literal name {a,b}.text. The reason is that in bash, brace expansion happens before parameter expansion.
A common trope on StackOverflow bash is: "Why doesn't x=99; echo {1..$x} work?"
The answer is "because braces are expanded before parameters/variables".
Therefore, I thought it should be possible to expand multiple variables using a single $ and a brace. I'd expect a=1; b=2; c=3; echo ${{a..c}} to print 1 2 3. First, the inner brace would expand to ${a} ${b} ${c} (which it does when writing echo \${{a..c}}). Then that result would undergo parameter expansion.
However, I got -bash: ${{a..c}}: bad substitution so {a..c} wasn't expanded at all.
Bash's manual is a bit more specific (emphasis mine).
Expansion is performed on the command line after it has been split into tokens [...]
The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion); word splitting; and filename expansion.
Note the ; and , in that list. "Left-to-right fashion" seems to apply to the whole (therefore unordered) list before the ;. Just like the mathematical operators * and / have no precedence over each other.
Ok, so brace expansion is not really of higher precedence than parameter expansion. It's just that both {1..$x} and ${{a..c}} are evaluated from left to right, meaning the brace { comes before the parameter $x and the parameter ${ comes before the brace {a..c}.
Or so I thought. However, when using $ instead of ${ then parameters on the left expand after braces on the right:
# in bash 5.0.3(1)
x=nil; x1=one; x2=two
echo ${x{1..2}} # prints `-bash: ${x{1..2}}: bad substitution`
echo $x{1..2} # prints `one two`
Question
Could it be that the bash manual is flawed or did I read it wrong?
If the manual is flawed: What is the exact order of all expansions?
I'm just asking because I'm curious. I don't plan to use thinks like $x{1..2} anywhere. I'm not interested in better solutions or alternatives to address multiple variables (e.g. array slices ${array[#]:1:2}). I just want to get a deeper understanding.
from: https://www.gnu.org/software/bash/manual/html_node/Brace-Expansion.html
To avoid conflicts with parameter expansion, the string ‘${’ is not
considered eligible for brace expansion, and inhibits brace expansion
until the closing ‘}’.
That said, for echo $x{1..2} , first the brace expansion takes place, and then the parameter expansion, so we have echo $x1 $x2. For echo ${x{1..2}} the brace expansion doesn't happen, because we are after the ${ and haven't reached the closing } of the parameter expansion.
Regarding the bash manual part you have quoted, left-to-right order still exists for the expansions (with respect to allowed nested ones). Things get clearer if you format the list instead of using , and ;:
brace expansion
In a left-to-right fashion:
tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution
word splitting
filename expansion.
Read Mo Budlong's 1988 classic Command Line Psychology, which was written for regular Unix, but most of it still applies to bash. The order of evaluation goes:
1 History substitution (except for the Bourne shell)
2 Splitting words, including special characters
3 Updating the history list (except for the Bourne shell)
4 Interpreting single and double quotes
5 Alias substitution (except for the Bourne shell)
6 Redirection of input and output (< > and |)
7 Variable substitution (variables starting with $)
8 Command substitution (commands inside back quotes)
9 File name expansion (file name wild cards)
So what bash does with code like {1..3} happens before step 7 above, and that's why the OP code fails.
But if we must, there's always eval, (which should only be used if the variables are known in advance, or first cautiously type checked):
a=1; b=2; c=3; eval echo \{$a..$c}
Output:
1 2 3
I got the question when i looking other's shell script.
I saw that declared
APP_NAME="${0##*[\\/]}"
Since I can't find any answer, what's the meaning of this code?
It's Shell Parameter Expansion to get script name itself, without path
See bash manual:
${parameter##word}
The word is expanded to produce a pattern and matched according to the rules described below (see Pattern Matching).
If the pattern matches the beginning of the expanded value of parameter, then the result of the expansion is the expanded value of parameter with the shortest matching pattern (the ‘#’ case) or the longest matching pattern (the ‘##’ case) deleted.
Pattern Matching:
*
Matches any string, including the null string. When the globstar shell option is enabled, and ‘*’ is used in a filename expansion context
[…]
Matches any one of the enclosed characters.
Explanation
${parameter<...>} expression means that you can expand shell parameters.
I.e. ${1:-"default_arg_value"} will be expanded to "default_arg_value" if script running without arguments.
0 - is a 0th argument, i.e. script name itself
${0##<pattern>} will delete longest matching to <pattern> part of $0
*[\\/] means any string that ends with \ or / symbol.
So, APP_NAME="${0##*[\\/]}" means that $APP_NAME will be initialized by script name itself, without path.
Sample
Let's suppose you have script a/b/c/d/test.sh:
#!/bin/bash
echo "${0##*[\/]}"
echo "${1#*[\/]}"
echo "${2##[\/]}"
$ bash a/b/c/d/test.sh /tmp/e /tmp/ff
> test.sh
> tmp/e
> tmp/ff
I am new to shell scripting:
I have following:
old=/dev/sda
new=/dev/sda5
Given these variables, I need to extract 5 from "new" string.
How should I go about it? sed? awk?
Tried using:
partitionno=$(echo $new | sed 's/$old//g')
To get the correct result with the least change to your command, try:
partitionno=$(echo "$new" | sed "s|$old||g")
There are two key points here:
Shell variables are not expanded inside single quotes. So '$old' remains as the original four characters: $, o, l, and d. For the shell variables to be expanded, use double quotes.
sed "s/$old//g" still won't work because there are too many slashes. A substitute command uses three slashes. After the shell expands $old, there are five slashes. The solution is to use a different delimiter for the substitute command. I chose | above because | is not likely to be in a file name.
Using Shell Parameter Expansion:
$ old=/dev/sda
$ new=/dev/sda5
$ echo "${new#$old}"
5
${parameter#word}
${parameter##word}
The word is expanded to produce a pattern just as in filename expansion (see Filename Expansion). If the pattern matches the beginning of the expanded value of parameter, then the result of the expansion is the expanded value of parameter with the shortest matching pattern (the ‘#’ case) or the longest matching pattern (the ‘##’ case) deleted. If parameter is ‘#’ or ‘’, the pattern removal operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with ‘#’ or ‘’, the pattern removal operation is applied to each member of the array in turn, and the expansion is the resultant list.