Parameterized substitutions (${foo%bar}, ${foo-bar}, etc) without using eval - bash

From envsubst man:
These substitutions are a subset of the substitutions that a shell
performs on unquoted and double-quoted strings. Other kinds of
substitutions done by a shell, such as ${variable-default} or
$(command-list) or `command-list`, are not performed by the envsubst
program, due to security reasons.
I'd like to perform variable substitution a string, supporting constructs like ${variable-default} or ${variable%suffix}. I don't want to allow running commands.
Apparently it's not possible using envsubst, on the other hand eval has serious security implications.
Is there some other possibility than writing custom interpolation function?

bash 4.4 introduced a new type of parameter expansion which might do what you want. Namely, ${foo#P} expands the value of foo as if it were a prompt string, and a prompt string does undergo a round of expansion just prior to being displayed.
${parameter#operator}
Parameter transformation. The expansion is either a transforma-
tion of the value of parameter or information about parameter
itself, depending on the value of operator. Each operator is a
single letter:
Q The expansion is a string that is the value of parameter
quoted in a format that can be reused as input.
E The expansion is a string that is the value of parameter
with backslash escape sequences expanded as with the
$'...' quoting mechansim.
P The expansion is a string that is the result of expanding
the value of parameter as if it were a prompt string (see
PROMPTING below).
A The expansion is a string in the form of an assignment
statement or declare command that, if evaluated, will
recreate parameter with its attributes and value.
a The expansion is a string consisting of flag values rep-
resenting parameter's attributes.
A quick example:
$ foo='${bar:-9}'
$ echo "$foo"
${bar:-9}
$ echo "${foo#P}"
9
$ bar=3
echo "${foo#P}"
3
It does, however, still allow running arbitrary commands via $(...):
$ foo='$(echo hi)'
$ echo "${foo#P}"
hi
Another caveat: it does, of course, also expand prompt escapes, so you may be more expansions than you expected if your string already contains some backslashes. There is some conflict between prompt escapes and escapes intended for echo -e.

Related

Bash 4.4 prompt escape for number of jobs currently running

I stumbled across this post where user chepner proposed in his answer the usage of \j (as mentioned in the bash manual) to retrieve the current running count of background jobs.
Basically it boils down to
num_jobs="\j"
echo ${num_jobs#P}
Can anyone enlighten me on what is going on here exactly? E.g.
why ${\j#P} is not working and
what #P is doing exactly?
Like any parameter expansion, you have to supply the name of a parameter, not an arbitrary string. \j isn't the name of a parameter; it's the text you want to get from a parameter expansion.
After the parameter has been expanded, #P further subjects the result to prompt expansion, so that \j is replaced by the number of jobs.
$ num_jobs="\j"
$ echo "${num_jobs}"
\j
$ echo "${num_jobs#P}"
0
The part before the # is the name of the parameter you're trying to expand, it can't be a string you want to modify somehow. And #P is a parameter expansion introduced in Bash 4.4 (see manual):
${parameter#operator}
The expansion is either a transformation of the value of parameter
or information about parameter itself, depending on the value of
operator. Each operator is a single letter:
P
The expansion is a string that is the result of expanding the value of
parameter as if it were a prompt string (see Controlling the Prompt).

Are quotes necessary in bash when declaring local variables based on the command line argument variable expansion? [duplicate]

This question already has answers here:
Quoting vs not quoting the variable on the RHS of a variable assignment
(5 answers)
Closed 4 years ago.
Are the quotes in the below example necessary or superfluous. And why?
#!/bin/bash
arg1="$1"
arg2="$2"
How do you explain the fact when $1 is 123 echo abc, the first assignment is not interpreted as:
arg1=123 echo abc
which is a normal command (echo) call with argument abc and an environment variable (arg) passed to the execution.
From section 2.9.1 of the POSIX shell syntax specification:
Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value.
String-splitting and globbing (the steps which double quotes suppress) are not in this list.
Thus, the quotes are superfluous -- not just for assignments where the right-and side refers to a positional parameter, but for all assignments barring those where (1) the behavior of single-quoted, not double-quoted, strings are desired; or (2) whitespace or other content in the value would be otherwise parsed as syntactic rather than literal.
(Note that the decision on how to parse a command -- thus, whether it is an assignment, a simple command, a compound command, or something else -- takes place before parameter expansions; thus, var=$1 is determined to be an assignment before the value of $1 is ever considered! Were this untrue, such that data could silently become syntax, it would be far more difficult -- if not impossible -- to write secure code handling untrusted data in bash).

Why does field splitting not occur after parameter expansion in an assignment statement in shell?

Consider the following two assignments.
$ a="foo bar"
$ b=$a
$ b=foo bar
bash: bar: command not found
Why does the second assignment work fine? How is the second command any different from the third command?
I was hoping the second assignment to fail because
b=$a
would expand to
b=foo bar
Since $a is not within double-quotes, foo bar is not quoted, therefore field-splitting should occur (as per my understanding) which would result in b=foo to be considered an assignment and bar to be a command that cannot be found.
Summary: I was expecting the 2nd command to fail for the same reason that caused the 3rd command to fail. Why does the 2nd command succeed?
I went through the POSIX but I am unable to find anything that specifies that field splitting won't occur after parameter expansion that occurs in an assignment.
I mean anywhere else field splitting would occur for an unquoted parameter after parameter expansion. For example,
$ a="foo bar"
$ printf "[%s] [%s]\n" $a
[foo] [bar]
See Section 2.6.5.
After parameter expansion (Parameter Expansion), command substitution (Command Substitution), and arithmetic expansion (Arithmetic Expansion), the shell shall scan the results of expansions and substitutions that did not occur in double-quotes for field splitting and multiple fields can result.
So which part of the POSIX standard prevents field splitting when parameter expansion occurs in an assignment statement?
In 2.9.1, "Simple Commands":
The words that are recognized as variable assignments or redirections according to Shell Grammar Rules are saved for processing in steps 3 and 4.
Step 2 -- which is explicitly skipped in this case per the above text -- reiterates that it ignores assignments when performing expansion and field splitting:
The words that are not variable assignments or redirections shall be expanded. If any fields remain following their expansion, the first field shall be considered the command name and remaining fields are the arguments for the command.
Thus, it's step 2 that determines the command to run (based on contents other than variable assignments and redirections), which addresses the b=$a case given in your question.
Step 4 performs other expansions -- "tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal" -- for assignments. Notably, field splitting is not a member of this set. Indeed, it's explicit in 2.6 that none of these create multiple words in and of themselves:
Tilde expansions, parameter expansions, command substitutions, arithmetic expansions, and quote removals that occur within a single word expand to a single field. It is only field splitting or pathname expansion that can create multiple fields from a single word. The single exception to this rule is the expansion of the special parameter '#' within double-quotes, as described in Special Parameters.

Command substitution and field splitting in shell

I understand why the following command fails.
$ a=foo bar
-bash: bar: command not found
It attempts to first execute a=foo and then execute bar which fails because there is no such command called bar.
But I don't understand why this works. I was expecting the following command to fail as well.
$ a=$(echo foo bar)
$ echo "$a"
foo bar
As per http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_05 first command substitution happens, and then field splitting happens.
2.6 Word Expansions
This section describes the various expansions that are performed on
words. Not all expansions are performed on every word, as explained in
the following sections.
Tilde expansions, parameter expansions, command substitutions,
arithmetic expansions, and quote removals that occur within a single
word expand to a single field. It is only field splitting or pathname
expansion that can create multiple fields from a single word. The
single exception to this rule is the expansion of the special
parameter '#' within double-quotes, as described in Special
Parameters.
The order of word expansion shall be as follows:
Tilde expansion (see Tilde Expansion), parameter expansion (see Parameter Expansion), command substitution (see Command Substitution),
and arithmetic expansion (see Arithmetic Expansion) shall be
performed, beginning to end. See item 5 in Token Recognition.
Field splitting (see Field Splitting) shall be performed on the portions of the fields generated by step 1, unless IFS is null.
Pathname expansion (see Pathname Expansion) shall be performed, unless set -f is in effect.
Quote removal (see Quote Removal) shall always be performed last.
So after command subsitution,
a=$(echo foo bar)
becomes
a=foo bar
And then after to field splitting, a=foo should be executed first and then bar should be executed and then we should have the same error, i.e. bar: command not found. Why does a=$(echo foo bar) work fine then?
The answer is in 2.9.1 Simple Commands I believe.
Specifically points 1 and 4:
 1. The words that are recognized as variable assignments or redirections according to Shell Grammar Rules are saved for processing in steps 3 and 4.
 4. Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value.
Or in the bash reference manual in 3.4 Shell Parameters:
A variable may be assigned to by a statement of the form
name=[value]
If value is not given, the variable is assigned the null string. All values undergo tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, and quote removal (detailed below).

using double quotes in bash export statement

Hello I'm reading a book about bash scripting and the author says to add the following to the end of my .bashrc file. export PATH=~/bin:"$PATH" in order to execute my file from the command line by typing its name. I notice however that if I put export PATH=~/bin:$PATH I can achieve the same result. So my question is what is the difference between the one with quotes and the one without quotes? thanks.
The quotes won't hurt anything, but neither are they necessary. Assignments are processed specially by the shell. From the man page:
A variable may be assigned to by a statement of the form
name=[value]
If value is not given, the variable is assigned the null string. All values undergo tilde expansion, parameter and variable
expansion, command substitution, arithmetic expansion, and
quote removal (see EXPANSION below).
Notice that word-splitting and pathname generation are not on the list in bold. These are the two types of expansion you are trying to prevent by quoting a parameter expansion, but in this context they are not performed. The same rules apply to the assignments that are passed to the export built-in command.
You must include the variable PATH inside double quotes. So that it would handle the filepaths which has spaces but without double quotes, it won't handle the filenames which has spaces in it.
I was facing the same with trying to assign a JSON string to a variable in the terminal.
Wrap it with Single Quotes or Double Quotes
Use single quotes, if you string contains double quotes and vice-versa.
$ export TEMP_ENV='I like the "London" bridge'
$ echo $TEMP_ENV
>> I like the "London" bridge
$ export TEMP_ENV="I like the 'London' bridge"
$ echo $TEMP_ENV
>> I like the 'London' bridge

Resources