GNU Bash - 3.6.6 Here Documents
[n]<<[-]word
here-document
delimiter
If any part of word is quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion, the character sequence \newline is ignored, and ‘\’ must be used to quote the characters ‘\’, ‘$’, and ‘`’.
If I single-quote EOF, it works. I think because bash /bin/bash process to be invoked gets un-expanded strings and then the invoked process interprets the lines.
$ /bin/bash<<'EOF'
#!/bin/bash
echo $BASH_VERSION
EOF
3.2.57(1)-release
However, the below is causing an error. I thought BASH_VERSION would have been expanded and the version of current bash process is passed to the /bin/bash process to be invoked. But not working.
$ /bin/bash<<EOF
#!/bin/bash
echo $BASH_VERSION
EOF
/bin/bash: line 2: syntax error near unexpected token `('
/bin/bash: line 2: `echo 5.0.17(1)-release'
/bin/bash<<EOF
#!/bin/bash
echo $BASH_VERSION
EOF
As you can infer from the error message, the heredoc is being expanded to:
/bin/bash<<EOF
#!/bin/bash
echo 5.0.17(1)-release
EOF
It sounds like that's what you expect: it's being expanded to the outer shell's version. The problem isn't with the heredoc or the expansion; it's that unquoted parentheses are a syntax error. Try running just the echo command by hand and you'll get the same error:
$ echo 5.0.17(1)-release
bash: syntax error near unexpected token `('
To fix this, you could add extra quotes:
/bin/bash<<EOF
echo '$BASH_VERSION'
EOF
This will work and print the outer shell's version. I used single quotes to demonstrate that these quotes will not inhibit variable expansion. The outer shell doesn't see these quotes. Only the inner shell does.
(I also got rid of the #!/bin/bash shebang line. There's no need for it since you're explicitly invoking bash.)
However, quoting is not 100% robust. If $BASH_VERSION happened to contain single quotes you'd have a problem. The quotes make parentheses ( ) safe but they aren't foolproof. As a general technique, if you want this to be completely safe no matter what special characters are in play you'll have to jump through some ugly hoops.
Use printf '%q' to escape all special characters.
/bin/bash <<EOF
echo $(printf '%q' "$BASH_VERSION")
EOF
This will expand to echo 5.0.17\(1\)-release.
Pass it in as an environment variable and use <<'EOF' to disable interpolation inside the script.
OUTER_VERSION="$BASH_VERSION" /bin/bash <<'EOF'
echo "$OUTER_VERSION"
EOF
This would be my choice. I prefer use the <<'EOF' form whenever possible. Having the parent shell interpolate the script being passed to a child shell can be confusing and difficult to reason about. Also, the explicit $OUTER_VERSION variable makes it crystal clear what's happening.
Use bash -c 'script' instead of a heredoc and then pass the version in as a command-line argument.
bash -c 'echo "$1"' bash "$BASH_VERSION"
I might go with this for a single-line script.
If you don't quote EOF, variables in the heredoc are expanded by the original shell before passing it as input to the invoked shell. So it's equivalent to executing
echo 3.2.57(1)-release
in the invoked shell. That's not valid bash syntax, so you get an error.
Quoting the word prevents variable expansion, so the invoked shell receives $BASH_VERSION literally, and expands it itself.
In the first case, the quotes prevent any changes in the here document, so the sub-shell sees echo $BASH_VERSION and it expands the string and echoes it.
In the second case, the absence of quotes means that the first shell expands the information and it sees echo 3.2.57(1)-release, and if you type that at the command line, you get the syntax error.
If you used echo "$BASH_VERSION" in both, then both would work, but different shells would expand $BASH_VERSION.
Related
What is the correct way to call some command stored in variable?
Are there any differences between 1 and 2?
#!/bin/sh
cmd="ls -la $APPROOTDIR | grep exception"
#1
$cmd
#2
eval "$cmd"
Unix shells operate a series of transformations on each line of input before executing them. For most shells it looks something like this (taken from the Bash man page):
initial word splitting
brace expansion
tilde expansion
parameter, variable and arithmetic expansion
command substitution
secondary word splitting
path expansion (aka globbing)
quote removal
Using $cmd directly gets it replaced by your command during the parameter expansion phase, and it then undergoes all following transformations.
Using eval "$cmd" does nothing until the quote removal phase, where $cmd is returned as is, and passed as a parameter to eval, whose function is to run the whole chain again before executing.
So basically, they're the same in most cases and differ when your command makes use of the transformation steps up to parameter expansion. For example, using brace expansion:
$ cmd="echo foo{bar,baz}"
$ $cmd
foo{bar,baz}
$ eval "$cmd"
foobar foobaz
If you just do eval $cmd when we do cmd="ls -l" (interactively and in a script), you get the desired result. In your case, you have a pipe with a grep without a pattern, so the grep part will fail with an error message. Just $cmd will generate a "command not found" (or some such) message.
So try use to eval (near "The args are read and concatenated together") and use a finished command, not one that generates an error message.
$cmd would just replace the variable with it's value to be executed on command line.
eval "$cmd" does variable expansion & command substitution before executing the resulting value on command line
The 2nd method is helpful when you wanna run commands that aren't flexible eg.
for i in {$a..$b} format loop won't work because it doesn't allow variables. In this case, a pipe to bash or eval is a workaround.
Tested on Mac OSX 10.6.8, Bash 3.2.48
I think you should put
`
(backtick) symbols around your variable.
I'm trying to interpolate variables inside of a bash heredoc:
var=$1
sudo tee "/path/to/outfile" > /dev/null << "EOF"
Some text that contains my $var
EOF
This isn't working as I'd expect ($var is treated literally, not expanded).
I need to use sudo tee because creating the file requires sudo. Doing something like:
sudo cat > /path/to/outfile <<EOT
my text...
EOT
Doesn't work, because >outfile opens the file in the current shell, which is not using sudo.
In answer to your first question, there's no parameter substitution because you've put the delimiter in quotes - the bash manual says:
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter expansion, command substitution, arithmetic expansion, or
pathname expansion is performed on word. If any characters in word are
quoted, the delimiter is the result of quote removal on word, and the
lines in the here-document are not expanded. If word is unquoted, all
lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion. [...]
If you change your first example to use <<EOF instead of << "EOF" you'll find that it works.
In your second example, the shell invokes sudo only with the parameter cat, and the redirection applies to the output of sudo cat as the original user. It'll work if you try:
sudo sh -c "cat > /path/to/outfile" <<EOT
my text...
EOT
Don't use quotes with <<EOF:
var=$1
sudo tee "/path/to/outfile" > /dev/null <<EOF
Some text that contains my $var
EOF
Variable expansion is the default behavior inside of here-docs. You disable that behavior by quoting the label (with single or double quotes).
As a late corolloary to the earlier answers here, you probably end up in situations where you want some but not all variables to be interpolated. You can solve that by using backslashes to escape dollar signs and backticks; or you can put the static text in a variable.
Name='Rich Ba$tard'
dough='$$$dollars$$$'
cat <<____HERE
$Name, you can win a lot of $dough this week!
Notice that \`backticks' need escaping if you want
literal text, not `pwd`, just like in variables like
\$HOME (current value: $HOME)
____HERE
Demo: https://ideone.com/rMF2XA
Note that any of the quoting mechanisms -- \____HERE or "____HERE" or '____HERE' -- will disable all variable interpolation, and turn the here-document into a piece of literal text.
A common task is to combine local variables with script which should be evaluated by a different shell, programming language, or remote host.
local=$(uname)
ssh -t remote <<:
echo "$local is the value from the host which ran the ssh command"
# Prevent here doc from expanding locally; remote won't see backslash
remote=\$(uname)
# Same here
echo "\$remote is the value from the host we ssh:ed to"
:
I'm writing a shell, and am getting unexpected parsing from both bash, dash, and busybox's ash:
echo "`echo a #`"
prints a, however
echo "$(echo a #)"
gives an error about missing a closing ).
How is a comment in a command-substitution parsed according to POSIX?
So, for the commands:
echo "`echo a #`"
and
echo "$(echo a #)"
Will the shell parse the comment as extending to the end of the command substitution, or to the end of the line?
Also, will the shell parse it differently if the command substitutions are not in double quotes?
Finally, are there any other constructs (either in POSIX or bash) where a comment can start inside quotes like this?
According to Posix (Shell&Utilities, §2.6.3), "`echo a #`" is undefined (implying that it should not be used):
The search for the matching backquote shall be satisfied by the first unquoted non-escaped backquote; during this search, if a non-escaped backquote is encountered within a shell comment, … undefined results occur. (emphasis added)
However, the $( command substitution marker is terminated by the "first matching )"; the implication (made explicit by examples in the Rationale, Note 1) is that the matching ) cannot be inside of a shell comment, here-doc or quoted string.
The quotes surrounding the command substitution are not relevant in either case (although, of course, "undefined results" could be different in the quoted case, since they are undefined.)
In bash and certain other shells, comments could also be present inside process substitution (eg., <(…)); however, process substitution cannot be quoted.
Notes:
Thanks to #mklement0, who included this link in a comment.
What is the correct way to call some command stored in variable?
Are there any differences between 1 and 2?
#!/bin/sh
cmd="ls -la $APPROOTDIR | grep exception"
#1
$cmd
#2
eval "$cmd"
Unix shells operate a series of transformations on each line of input before executing them. For most shells it looks something like this (taken from the Bash man page):
initial word splitting
brace expansion
tilde expansion
parameter, variable and arithmetic expansion
command substitution
secondary word splitting
path expansion (aka globbing)
quote removal
Using $cmd directly gets it replaced by your command during the parameter expansion phase, and it then undergoes all following transformations.
Using eval "$cmd" does nothing until the quote removal phase, where $cmd is returned as is, and passed as a parameter to eval, whose function is to run the whole chain again before executing.
So basically, they're the same in most cases and differ when your command makes use of the transformation steps up to parameter expansion. For example, using brace expansion:
$ cmd="echo foo{bar,baz}"
$ $cmd
foo{bar,baz}
$ eval "$cmd"
foobar foobaz
If you just do eval $cmd when we do cmd="ls -l" (interactively and in a script), you get the desired result. In your case, you have a pipe with a grep without a pattern, so the grep part will fail with an error message. Just $cmd will generate a "command not found" (or some such) message.
So try use to eval (near "The args are read and concatenated together") and use a finished command, not one that generates an error message.
$cmd would just replace the variable with it's value to be executed on command line.
eval "$cmd" does variable expansion & command substitution before executing the resulting value on command line
The 2nd method is helpful when you wanna run commands that aren't flexible eg.
for i in {$a..$b} format loop won't work because it doesn't allow variables. In this case, a pipe to bash or eval is a workaround.
Tested on Mac OSX 10.6.8, Bash 3.2.48
I think you should put
`
(backtick) symbols around your variable.
This question already has answers here:
What is the difference between $(command) and `command` in shell programming?
(6 answers)
Closed last year.
There are two ways to capture the output of command line in bash:
Legacy Bourne shell backticks ``:
var=`command`
$() syntax (which as far as I know is Bash specific, or at least not supported by non-POSIX old shells like original Bourne)
var=$(command)
Is there any benefit to using the second syntax compared to backticks? Or are the two fully 100% equivalent?
The major one is the ability to nest them, commands within commands, without losing your sanity trying to figure out if some form of escaping will work on the backticks.
An example, though somewhat contrived:
deps=$(find /dir -name $(ls -1tr 201112[0-9][0-9]*.txt | tail -1l) -print)
which will give you a list of all files in the /dir directory tree which have the same name as the earliest dated text file from December 2011 (a).
Another example would be something like getting the name (not the full path) of the parent directory:
pax> cd /home/pax/xyzzy/plugh
pax> parent=$(basename $(dirname $PWD))
pax> echo $parent
xyzzy
(a) Now that specific command may not actually work, I haven't tested the functionality. So, if you vote me down for it, you've lost sight of the intent :-) It's meant just as an illustration as to how you can nest, not as a bug-free production-ready snippet.
Suppose you want to find the lib directory corresponding to where gcc is installed. You have a choice:
libdir=$(dirname $(dirname $(which gcc)))/lib
libdir=`dirname \`dirname \\\`which gcc\\\`\``/lib
The first is easier than the second - use the first.
The backticks (`...`) is the legacy syntax required by only the very oldest of non-POSIX-compatible bourne-shells and $(...) is POSIX and more preferred for several reasons:
Backslashes (\) inside backticks are handled in a non-obvious manner:
$ echo "`echo \\a`" "$(echo \\a)"
a \a
$ echo "`echo \\\\a`" "$(echo \\\\a)"
\a \\a
# Note that this is true for *single quotes* too!
$ foo=`echo '\\'`; bar=$(echo '\\'); echo "foo is $foo, bar is $bar"
foo is \, bar is \\
Nested quoting inside $() is far more convenient:
echo "x is $(sed ... <<<"$y")"
instead of:
echo "x is `sed ... <<<\"$y\"`"
or writing something like:
IPs_inna_string=`awk "/\`cat /etc/myname\`/"'{print $1}' /etc/hosts`
because $() uses an entirely new context for quoting
which is not portable as Bourne and Korn shells would require these backslashes, while Bash and dash don't.
Syntax for nesting command substitutions is easier:
x=$(grep "$(dirname "$path")" file)
than:
x=`grep "\`dirname \"$path\"\`" file`
because $() enforces an entirely new context for quoting, so each command substitution is protected and can be treated on its own without special concern over quoting and escaping. When using backticks, it gets uglier and uglier after two and above levels.
Few more examples:
echo `echo `ls`` # INCORRECT
echo `echo \`ls\`` # CORRECT
echo $(echo $(ls)) # CORRECT
It solves a problem of inconsistent behavior when using backquotes:
echo '\$x' outputs \$x
echo `echo '\$x'` outputs $x
echo $(echo '\$x') outputs \$x
Backticks syntax has historical restrictions on the contents of the embedded command and cannot handle some valid scripts that include backquotes, while the newer $() form can process any kind of valid embedded script.
For example, these otherwise valid embedded scripts do not work in the left column, but do work on the rightIEEE:
echo ` echo $(
cat <<\eof cat <<\eof
a here-doc with ` a here-doc with )
eof eof
` )
echo ` echo $(
echo abc # a comment with ` echo abc # a comment with )
` )
echo ` echo $(
echo '`' echo ')'
` )
Therefore the syntax for $-prefixed command substitution should be the preferred method, because it is visually clear with clean syntax (improves human and machine readability), it is nestable and intuitive, its inner parsing is separate, and it is also more consistent (with all other expansions that are parsed from within double-quotes) where backticks are the only exception and ` character is easily camouflaged when adjacent to " making it even more difficult to read, especially with small or unusual fonts.
Source: Why is $(...) preferred over `...` (backticks)? at BashFAQ
See also:
POSIX standard section "2.6.3 Command Substitution"
POSIX rationale for including the $() syntax
Command Substitution
bash-hackers: command substitution
From man bash:
$(command)
or
`command`
Bash performs the expansion by executing command and replacing the com-
mand substitution with the standard output of the command, with any
trailing newlines deleted. Embedded newlines are not deleted, but they
may be removed during word splitting. The command substitution $(cat
file) can be replaced by the equivalent but faster $(< file).
When the old-style backquote form of substitution is used, backslash
retains its literal meaning except when followed by $, `, or \. The
first backquote not preceded by a backslash terminates the command sub-
stitution. When using the $(command) form, all characters between the
parentheses make up the command; none are treated specially.
In addition to the other answers,
$(...)
stands out visually better than
`...`
Backticks look too much like apostrophes; this varies depending on the font you're using.
(And, as I just noticed, backticks are a lot harder to enter in inline code samples.)
$() allows nesting.
out=$(echo today is $(date))
I think backticks does not allow it.
It is the POSIX standard that defines the $(command) form of command substitution. Most shells in use today are POSIX compliant and support this preferred form over the archaic backtick notation. The command substitution section (2.6.3) of the Shell Language document describes this:
Command substitution allows the output of a command to be substituted in place of the command name itself. Command substitution shall occur when the command is enclosed as follows:
$(command)
or (backquoted version):
`command`
The shell shall expand the command substitution by executing command
in a subshell environment (see Shell Execution Environment) and
replacing the command substitution (the text of command plus the
enclosing "$()" or backquotes) with the standard output of the
command, removing sequences of one or more <newline> characters at the
end of the substitution. Embedded <newline> characters before the end
of the output shall not be removed; however, they may be treated as
field delimiters and eliminated during field splitting, depending on
the value of IFS and quoting that is in effect. If the output contains
any null bytes, the behavior is unspecified.
Within the backquoted style of command substitution, <backslash> shall
retain its literal meaning, except when followed by: '$' , '`', or
<backslash>. The search for the matching backquote shall be satisfied
by the first unquoted non-escaped backquote; during this search, if a
non-escaped backquote is encountered within a shell comment, a
here-document, an embedded command substitution of the $(command)
form, or a quoted string, undefined results occur. A single-quoted or
double-quoted string that begins, but does not end, within the "`...`"
sequence produces undefined results.
With the $(command) form, all characters following the open
parenthesis to the matching closing parenthesis constitute the
command. Any valid shell script can be used for command, except a
script consisting solely of redirections which produces unspecified
results.
The results of command substitution shall not be processed for further
tilde expansion, parameter expansion, command substitution, or
arithmetic expansion. If a command substitution occurs inside
double-quotes, field splitting and pathname expansion shall not be
performed on the results of the substitution.
Command substitution can be nested. To specify nesting within the
backquoted version, the application shall precede the inner backquotes
with <backslash> characters; for example:
\`command\`
The syntax of the shell command language has an ambiguity for expansions beginning with "$((",
which can introduce an arithmetic expansion or a command substitution that starts with a subshell.
Arithmetic expansion has precedence; that is, the shell shall first determine
whether it can parse the expansion as an arithmetic expansion
and shall only parse the expansion as a command substitution
if it determines that it cannot parse the expansion as an arithmetic expansion.
The shell need not evaluate nested expansions when performing this determination.
If it encounters the end of input without already having determined
that it cannot parse the expansion as an arithmetic expansion,
the shell shall treat the expansion as an incomplete arithmetic expansion and report a syntax error.
A conforming application shall ensure that it separates the "$(" and '(' into two tokens
(that is, separate them with white space) in a command substitution that starts with a subshell.
For example, a command substitution containing a single subshell could be written as:
$( (command) )
I came up with a perfectly valid example of $(...) over `...`.
I was using a remote desktop to Windows running Cygwin and wanted to iterate over a result of a command. Sadly, the backtick character was impossible to enter, either due to the remote desktop thing or Cygwin itself.
It's sane to assume that a dollar sign and parentheses will be easier to type in such strange setups.
Here in 2021 it is worth mentioning a curious fact as a supplement to the other answers.
The Microsoft DevOps YAML "scripting" for pipelines may include Bash tasks. However, the notation $() is used for referring to variables defined in the YAML context, so in this case backticks should be used for capturing the output of commands.
This is mostly a problem when copying scripting code into a YAML script since the DevOps preprocessor is very forgiving about nonexisting variables, so there will not be any error message.