I've got a variable named formatted_deploy_history that has multi-lined content. I'm trying to pipe that content into a series of commands and save the end result to a variable. Here's what I've done:
nwhin_version="$(echo "${formatted_deploy_history}" | grep "productName" | grep "NHINC" | head -n1 | grep -o "[0-9]*")"
This works the way I want, but it seems like it should be a syntax error. Why is it that the second quote doesn't interfere with the rest of the command? I'd expect a syntax error because this would be interpreted like this:
nwhin_version="$(echo "
with trailing characters.
Your string contains command substitution ($(...)), which is its own world - delimited by $( and ).
Inside that world, you're free to use double quotes without escaping.
See section 2.3, "Token Recognition", in the POSIX Shell Command Language specification:
If the current character is an unquoted $ or `, the shell shall identify the start of any candidates for parameter expansion ( Parameter Expansion), command substitution ( Command Substitution), or arithmetic expansion ( Arithmetic Expansion) from their introductory unquoted character sequences: $ or ${, $( or ```, and "$((", respectively. The shell shall read sufficient input to determine the end of the unit to be expanded (as explained in the cited sections). While processing the characters, if instances of expansions or quoting are found nested within the substitution, the shell shall recursively process them in the manner specified for the construct that is found. The characters found from the beginning of the substitution to its end, allowing for any recursion necessary to recognize embedded constructs, shall be included unmodified in the result token, including any embedded or enclosing substitution operators or quotes. The token shall not be delimited by the end of the substitution.
I believe you don't need quotes around the command substitution.
Try this:
nwhin_version=$(echo "${formatted_deploy_history}" | grep "productName" | grep "NHINC" | head -n1 | grep -o "[0-9]*")
Related
I use this to make passwords:
echo $(LC_CTYPE=C < /dev/urandom tr -cd [:graph:] | tr -d '\n' | head -c32) # pwg ##1st
Varying length, graph to alnum, or whatever.
I don't understand the redirection being used. I understand the outer $() format but this
echo $(LC_CTYPE=C < /dev/urandom tr -cd [:graph:] ...
doesn't make sense.
The LC_CTYPE is setting the character type. Got that.
tr is converting characters to those in the set :graph:. Got that.
The echo is there to give a line-break.
But how can you redirecct urandom in to a env variable setting
expression. ?
How, does tr know to retrieve urandom from before it. ?
To me it seems it should be:
echo $(LC_CTYPE=C tr -cd [:graph:] < /dev/urandom | tr -d '\n' | head -c32) # pwg ##2nd
Which also works. I just can't understand why the first variant works?
Some info from the manual:
3.6 Redirections
Before a command is executed, its input and output may be redirected using a special notation interpreted by the shell... The following redirection operators may precede or appear anywhere within a simple command or may follow a command. ...
3.7.1 Simple Command Expansion
When a simple command is executed, the shell performs the following expansions, assignments, and redirections, from left to right.
The words that the parser has marked as variable assignments (those preceding the command name) and redirections are saved for later processing.
The words that are not variable assignments or redirections are expanded (see Shell Expansions). If any words remain after expansion, the first word is taken to be the name of the command and the remaining words are the arguments.
Redirections are performed as described above (see Redirections).
The text after the ‘=’ in each variable assignment undergoes tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal before being assigned to the variable.
< doesn't redirect to the thing before the < -- it just redirects to stdin. So the order doesn't really matter. Whereever it is in command line < /dev/urandom redirects stdin of the command to come from /dev/urandom. Before the command, after the command, before some arguments, after the arguments, doesn't matter. It just redirects stdin from the specified file or device.
I am trying to pass the "strings" from a file as input to grep using the -F (fixed string) parameter.
From grep the man page, the expected format is newline-separated:
-F, --fixed-strings
Interpret PATTERN as a list of fixed strings (instead of regular expressions), separated by newlines, any of which is to be matched.
How can this be done in bash? I have:
#!/bin/bash
INFILE=$1
DIR=$2
# Create a newline-separated string array
STRINGS="";
while read -r string; do
STRINGS+=$'\n'$string;
done < <(strings $INFILE);
cd $DIR
for file in *; do
grep -Frn \"$STRINGS\" .
done;
But grep reports error at run-time regarding input formatting. Grep is interpreting the passed string arguments as parameters -- hence the need to pass them as one large string literal.
Debugging bash with -xand passing the first parameter (INFILE) as the script itself gives:
+ grep -Frn '"' '#!/bin/bash' 'INFILE=$1' 'DIR=$2' [...]
Try the following:
#!/bin/bash
inFile=$1
dir=$2
# Read all lines output by `string` into a single variable using
# a command substitution, $(...).
# Note that the trailing newlines is trimmed, but grep still recognizes
# the last line.
strings="$(strings "$inFile")"
cd "$dir"
for file in *; do
grep -Frn "$strings" .
done
string outputs each string found in the target file on its own line, so you can use its output as-is, via a command substitution ($(...)).
On a side note: strings is used to extract strings from binary files, and strings are only included if they're at least 4 ASCII(!) characters long and are followed by a newline or NUL.
Note that while the POSIX spec for strings does mandate locale-awareness with respect to character interpretation, both GNU strings and BSD/macOS strings recognize 7-bit ASCII characters only.
If, by contrast, your search strings come from a text file from which you want to strip empty and blank lines, use strings="$(awk 'NF>0' "$inFile")"
Double-quote your variable references and command substitutions to ensure that their values are used as-is.
Do not use \" unless you want to pass a literal " char. to the target command - as opposed to an unquoted one that has syntactical meaning to the shell.
In your particular case, \"$STRINGS\" breaks down as follows:
An unquoted reference to variable $STRINGS - because the enclosing " are \-escaped and therefore literals.
The resulting string - "<value-of-$STRINGS>" - due to $STRINGS being unquoted, is then subject to word-splitting
(and globbing), i.e., split into multiple arguments by whitespace. As a result, because grep expects the search term(s) as a single argument, the command breaks.
Do not use all-uppercase shell variable names in order to avoid conflicts with environment variables and special shell variables.
I've tried to run this code:
lsof | grep "/file/path/"
If the file has this sign:
`
I get an error:
syntax error near unexpected token '`'
I've tried to escape this sign with \, but I got the same error.
Operation System: CentOS 6
The ` character (often called a "backquote") is interpreted specially by the shell, even when surrounded by quotes.
Specifically, `cmd args` means "execute the command cmd args and replace the backquoted string with the output from that command." If it is surrounded by quotes -- "`cmd args`" -- the replaced string will not undergo word-splitting or pathname-expansion. (That's usually what you want, so quoted backquotes are common.)
That syntax has been deprecated for a long time, but it is still accepted. New code should use $(cmd args) instead. As above, you usually want to avoid word-splitting and pathname-expanding the replaced text, so you'll normally see "$(cmd args)".
In short, if you want to put a literal backquote into an argument string, you should either use single quotes:
lsof | grep '/file/path`s/'
or \-escape the backquote:
lsof | grep "/file/path\`s/"
When I went to answer this question, I was going to use the ${} notation, as I've seen so many times on here that it's preferable to backticks.
However, when I tried
joulesFinal=${echo $joules2 \* $cpu | bc}
I got the message
-bash: ${echo $joules * $cpu | bc}: bad substitution
but
joulesFinal=`echo $joules2 \* $cpu | bc`
works fine. So what other changes do I need to make?
The `` is called Command Substitution and is equivalent to $() (parenthesis), while you are using ${} (curly braces).
So all of these expressions are equal and mean "interpret the command placed inside":
joulesFinal=`echo $joules2 \* $cpu | bc`
joulesFinal=$(echo $joules2 \* $cpu | bc)
# v v
# ( instead of { v
# ) instead of }
While ${} expressions are used for variable substitution.
Note, though, that backticks are deprecated, while $() is POSIX compatible, so you should prefer the latter.
From man bash:
Command substitution allows the output of a command to replace the
command name. There are two forms:
$(command)
or
`command`
Also, `` are more difficult to handle, you cannot nest them for example. See comments below and also Why is $(...) preferred over ... (backticks)?.
They behave slightly differently in a specific case:
$ echo "`echo \"test\" `"
test
$ echo "$(echo \"test\" )"
"test"
So backticks silently remove the double quotes.
${} refer to Shell parameter expansion. Manual link:https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
The ‘$’ character introduces parameter expansion, command
substitution, or arithmetic expansion. The parameter name or symbol to
be expanded may be enclosed in braces, which are optional but serve to
protect the variable to be expanded from characters immediately
following it which could be interpreted as part of the name.
When braces are used, the matching ending brace is the first ‘}’ not
escaped by a backslash or within a quoted string, and not within an
embedded arithmetic expansion, command substitution, or parameter
expansion.
FULLPATH=/usr/share/X11/test.conf_d/sk-synaptics.conf
echo ${FULLPATH##*/}
echo ${FILENAME##*.}
First echo will get filename. second will get file extension as per manual ${parameter##word} section.
$(command)
`command`
refer to command substitution.
Bash performs the expansion by executing command in a subshell
environment and replacing the command substitution with the standard
output of the command, with any trailing newlines deleted.
https://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html
This question already has answers here:
What is the difference between $(command) and `command` in shell programming?
(6 answers)
Closed last year.
There are two ways to capture the output of command line in bash:
Legacy Bourne shell backticks ``:
var=`command`
$() syntax (which as far as I know is Bash specific, or at least not supported by non-POSIX old shells like original Bourne)
var=$(command)
Is there any benefit to using the second syntax compared to backticks? Or are the two fully 100% equivalent?
The major one is the ability to nest them, commands within commands, without losing your sanity trying to figure out if some form of escaping will work on the backticks.
An example, though somewhat contrived:
deps=$(find /dir -name $(ls -1tr 201112[0-9][0-9]*.txt | tail -1l) -print)
which will give you a list of all files in the /dir directory tree which have the same name as the earliest dated text file from December 2011 (a).
Another example would be something like getting the name (not the full path) of the parent directory:
pax> cd /home/pax/xyzzy/plugh
pax> parent=$(basename $(dirname $PWD))
pax> echo $parent
xyzzy
(a) Now that specific command may not actually work, I haven't tested the functionality. So, if you vote me down for it, you've lost sight of the intent :-) It's meant just as an illustration as to how you can nest, not as a bug-free production-ready snippet.
Suppose you want to find the lib directory corresponding to where gcc is installed. You have a choice:
libdir=$(dirname $(dirname $(which gcc)))/lib
libdir=`dirname \`dirname \\\`which gcc\\\`\``/lib
The first is easier than the second - use the first.
The backticks (`...`) is the legacy syntax required by only the very oldest of non-POSIX-compatible bourne-shells and $(...) is POSIX and more preferred for several reasons:
Backslashes (\) inside backticks are handled in a non-obvious manner:
$ echo "`echo \\a`" "$(echo \\a)"
a \a
$ echo "`echo \\\\a`" "$(echo \\\\a)"
\a \\a
# Note that this is true for *single quotes* too!
$ foo=`echo '\\'`; bar=$(echo '\\'); echo "foo is $foo, bar is $bar"
foo is \, bar is \\
Nested quoting inside $() is far more convenient:
echo "x is $(sed ... <<<"$y")"
instead of:
echo "x is `sed ... <<<\"$y\"`"
or writing something like:
IPs_inna_string=`awk "/\`cat /etc/myname\`/"'{print $1}' /etc/hosts`
because $() uses an entirely new context for quoting
which is not portable as Bourne and Korn shells would require these backslashes, while Bash and dash don't.
Syntax for nesting command substitutions is easier:
x=$(grep "$(dirname "$path")" file)
than:
x=`grep "\`dirname \"$path\"\`" file`
because $() enforces an entirely new context for quoting, so each command substitution is protected and can be treated on its own without special concern over quoting and escaping. When using backticks, it gets uglier and uglier after two and above levels.
Few more examples:
echo `echo `ls`` # INCORRECT
echo `echo \`ls\`` # CORRECT
echo $(echo $(ls)) # CORRECT
It solves a problem of inconsistent behavior when using backquotes:
echo '\$x' outputs \$x
echo `echo '\$x'` outputs $x
echo $(echo '\$x') outputs \$x
Backticks syntax has historical restrictions on the contents of the embedded command and cannot handle some valid scripts that include backquotes, while the newer $() form can process any kind of valid embedded script.
For example, these otherwise valid embedded scripts do not work in the left column, but do work on the rightIEEE:
echo ` echo $(
cat <<\eof cat <<\eof
a here-doc with ` a here-doc with )
eof eof
` )
echo ` echo $(
echo abc # a comment with ` echo abc # a comment with )
` )
echo ` echo $(
echo '`' echo ')'
` )
Therefore the syntax for $-prefixed command substitution should be the preferred method, because it is visually clear with clean syntax (improves human and machine readability), it is nestable and intuitive, its inner parsing is separate, and it is also more consistent (with all other expansions that are parsed from within double-quotes) where backticks are the only exception and ` character is easily camouflaged when adjacent to " making it even more difficult to read, especially with small or unusual fonts.
Source: Why is $(...) preferred over `...` (backticks)? at BashFAQ
See also:
POSIX standard section "2.6.3 Command Substitution"
POSIX rationale for including the $() syntax
Command Substitution
bash-hackers: command substitution
From man bash:
$(command)
or
`command`
Bash performs the expansion by executing command and replacing the com-
mand substitution with the standard output of the command, with any
trailing newlines deleted. Embedded newlines are not deleted, but they
may be removed during word splitting. The command substitution $(cat
file) can be replaced by the equivalent but faster $(< file).
When the old-style backquote form of substitution is used, backslash
retains its literal meaning except when followed by $, `, or \. The
first backquote not preceded by a backslash terminates the command sub-
stitution. When using the $(command) form, all characters between the
parentheses make up the command; none are treated specially.
In addition to the other answers,
$(...)
stands out visually better than
`...`
Backticks look too much like apostrophes; this varies depending on the font you're using.
(And, as I just noticed, backticks are a lot harder to enter in inline code samples.)
$() allows nesting.
out=$(echo today is $(date))
I think backticks does not allow it.
It is the POSIX standard that defines the $(command) form of command substitution. Most shells in use today are POSIX compliant and support this preferred form over the archaic backtick notation. The command substitution section (2.6.3) of the Shell Language document describes this:
Command substitution allows the output of a command to be substituted in place of the command name itself. Command substitution shall occur when the command is enclosed as follows:
$(command)
or (backquoted version):
`command`
The shell shall expand the command substitution by executing command
in a subshell environment (see Shell Execution Environment) and
replacing the command substitution (the text of command plus the
enclosing "$()" or backquotes) with the standard output of the
command, removing sequences of one or more <newline> characters at the
end of the substitution. Embedded <newline> characters before the end
of the output shall not be removed; however, they may be treated as
field delimiters and eliminated during field splitting, depending on
the value of IFS and quoting that is in effect. If the output contains
any null bytes, the behavior is unspecified.
Within the backquoted style of command substitution, <backslash> shall
retain its literal meaning, except when followed by: '$' , '`', or
<backslash>. The search for the matching backquote shall be satisfied
by the first unquoted non-escaped backquote; during this search, if a
non-escaped backquote is encountered within a shell comment, a
here-document, an embedded command substitution of the $(command)
form, or a quoted string, undefined results occur. A single-quoted or
double-quoted string that begins, but does not end, within the "`...`"
sequence produces undefined results.
With the $(command) form, all characters following the open
parenthesis to the matching closing parenthesis constitute the
command. Any valid shell script can be used for command, except a
script consisting solely of redirections which produces unspecified
results.
The results of command substitution shall not be processed for further
tilde expansion, parameter expansion, command substitution, or
arithmetic expansion. If a command substitution occurs inside
double-quotes, field splitting and pathname expansion shall not be
performed on the results of the substitution.
Command substitution can be nested. To specify nesting within the
backquoted version, the application shall precede the inner backquotes
with <backslash> characters; for example:
\`command\`
The syntax of the shell command language has an ambiguity for expansions beginning with "$((",
which can introduce an arithmetic expansion or a command substitution that starts with a subshell.
Arithmetic expansion has precedence; that is, the shell shall first determine
whether it can parse the expansion as an arithmetic expansion
and shall only parse the expansion as a command substitution
if it determines that it cannot parse the expansion as an arithmetic expansion.
The shell need not evaluate nested expansions when performing this determination.
If it encounters the end of input without already having determined
that it cannot parse the expansion as an arithmetic expansion,
the shell shall treat the expansion as an incomplete arithmetic expansion and report a syntax error.
A conforming application shall ensure that it separates the "$(" and '(' into two tokens
(that is, separate them with white space) in a command substitution that starts with a subshell.
For example, a command substitution containing a single subshell could be written as:
$( (command) )
I came up with a perfectly valid example of $(...) over `...`.
I was using a remote desktop to Windows running Cygwin and wanted to iterate over a result of a command. Sadly, the backtick character was impossible to enter, either due to the remote desktop thing or Cygwin itself.
It's sane to assume that a dollar sign and parentheses will be easier to type in such strange setups.
Here in 2021 it is worth mentioning a curious fact as a supplement to the other answers.
The Microsoft DevOps YAML "scripting" for pipelines may include Bash tasks. However, the notation $() is used for referring to variables defined in the YAML context, so in this case backticks should be used for capturing the output of commands.
This is mostly a problem when copying scripting code into a YAML script since the DevOps preprocessor is very forgiving about nonexisting variables, so there will not be any error message.