I have noticed several distinction between them:
Inside a <<EOF heredoc, new values can not be assigned to variables:
bash <<EOF
s=fds
echo $s
EOF
will print empty line, where
bash <<\EOF
s=fds
echo $s
EOF
will print the value of the variable s.
Global variables can be accessed within <<EOF but not within <<\EOF (with export it is possible to access variables inside <<\EOF):
s=fds
bash <<EOF
echo $s
EOF
will print the value fds, where,
s=fds
bash <<\EOF
echo $s
EOF
will print empty line.
So what are the differences between them and what is the legitimate documented behavior?
From the POSIX spec:
If any character in word is quoted, the delimiter shall be formed by performing quote removal on word, and the here-document lines shall not be expanded. Otherwise, the delimiter shall be the word itself.
So the <<EOF version has the shell expand all variables before running the here doc contents and the <<\EOF (or <<'EOF' or <<EO'F' etc.) versions don't expand the contents (which lets bash in this case do that work).
Try it with cat instead of bash for a clearer view on what is happening.
Also with printf '[%s]\n' "$s" and/or possibly bash -x instead of bash:
$ bash -x <<EOF
s=fds
printf '[%s]\n' "$s"
EOF
+ s=fds
+ printf '[%s]\n' ''
[]
$ bash -x <<\EOF
s=fds
printf '[%s]\n' "$s"
EOF
+ s=fds
+ printf '[%s]\n' fds
[fds]
Documentation: http://www.gnu.org/software/bash/manual/bash.html#Here-Documents
In your first example the delimiter is unquoted, so variable expansion occurs and it's like you're running the code
echo "s=fds
echo $s" | bash
which expands $s in the current shell, where it's empty. So the new shell sees
s=fds
echo
Read the Advanced Bash Scripting Guide & bash reference manual in particular about redirections:
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic
expansion, or filename expansion is performed on word. If any
characters in word are quoted, the delimiter is the result of quote
removal on word, and the lines in the here-document are not expanded.
If word is unquoted, all lines of the here-document are subjected to
parameter expansion, command substitution, and arithmetic expansion,
the character sequence \newline is ignored, and ‘\’ must be used to
quote the characters ‘\’, ‘$’, and ‘`’.
Related
Is it possible to create a heredoc that does not become subject to variable expansion?
e.g.
cat <<-EOF > somefile.sh
Do not print current value of $1 instead evaluate it later.
EOF
Update I am aware of escaping by \. My actual heredoc has many variables in it - and it is error prone and tedious to escape all of them.
Quote the delimiter:
cat <<-"EOF" > somefile.sh
Do not print current value of $1 instead evaluate it later.
EOF
This results in:
$ cat somefile.sh
Do not print current value of $1 instead evaluate it later.
Documentation
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word. If
any characters in word are quoted, the delimiter is the result of
quote removal on word, and the lines in the here-document are
not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command
substitution, and arithmetic expansion, the character sequence
\ is ignored, and \ must be used to quote the characters \,
$, and `.
If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing
delimiter. This allows here-documents within shell scripts to be
indented in a natural fashion. [Emphasis added.]
Put backlash before the $ sign
$ VAR=XXX
$ cat << END
> dk
> \$VAR
> END
dk
$VAR
GNU Bash - 3.6.6 Here Documents
[n]<<[-]word
here-document
delimiter
If any part of word is quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion, the character sequence \newline is ignored, and ‘\’ must be used to quote the characters ‘\’, ‘$’, and ‘`’.
If I single-quote EOF, it works. I think because bash /bin/bash process to be invoked gets un-expanded strings and then the invoked process interprets the lines.
$ /bin/bash<<'EOF'
#!/bin/bash
echo $BASH_VERSION
EOF
3.2.57(1)-release
However, the below is causing an error. I thought BASH_VERSION would have been expanded and the version of current bash process is passed to the /bin/bash process to be invoked. But not working.
$ /bin/bash<<EOF
#!/bin/bash
echo $BASH_VERSION
EOF
/bin/bash: line 2: syntax error near unexpected token `('
/bin/bash: line 2: `echo 5.0.17(1)-release'
/bin/bash<<EOF
#!/bin/bash
echo $BASH_VERSION
EOF
As you can infer from the error message, the heredoc is being expanded to:
/bin/bash<<EOF
#!/bin/bash
echo 5.0.17(1)-release
EOF
It sounds like that's what you expect: it's being expanded to the outer shell's version. The problem isn't with the heredoc or the expansion; it's that unquoted parentheses are a syntax error. Try running just the echo command by hand and you'll get the same error:
$ echo 5.0.17(1)-release
bash: syntax error near unexpected token `('
To fix this, you could add extra quotes:
/bin/bash<<EOF
echo '$BASH_VERSION'
EOF
This will work and print the outer shell's version. I used single quotes to demonstrate that these quotes will not inhibit variable expansion. The outer shell doesn't see these quotes. Only the inner shell does.
(I also got rid of the #!/bin/bash shebang line. There's no need for it since you're explicitly invoking bash.)
However, quoting is not 100% robust. If $BASH_VERSION happened to contain single quotes you'd have a problem. The quotes make parentheses ( ) safe but they aren't foolproof. As a general technique, if you want this to be completely safe no matter what special characters are in play you'll have to jump through some ugly hoops.
Use printf '%q' to escape all special characters.
/bin/bash <<EOF
echo $(printf '%q' "$BASH_VERSION")
EOF
This will expand to echo 5.0.17\(1\)-release.
Pass it in as an environment variable and use <<'EOF' to disable interpolation inside the script.
OUTER_VERSION="$BASH_VERSION" /bin/bash <<'EOF'
echo "$OUTER_VERSION"
EOF
This would be my choice. I prefer use the <<'EOF' form whenever possible. Having the parent shell interpolate the script being passed to a child shell can be confusing and difficult to reason about. Also, the explicit $OUTER_VERSION variable makes it crystal clear what's happening.
Use bash -c 'script' instead of a heredoc and then pass the version in as a command-line argument.
bash -c 'echo "$1"' bash "$BASH_VERSION"
I might go with this for a single-line script.
If you don't quote EOF, variables in the heredoc are expanded by the original shell before passing it as input to the invoked shell. So it's equivalent to executing
echo 3.2.57(1)-release
in the invoked shell. That's not valid bash syntax, so you get an error.
Quoting the word prevents variable expansion, so the invoked shell receives $BASH_VERSION literally, and expands it itself.
In the first case, the quotes prevent any changes in the here document, so the sub-shell sees echo $BASH_VERSION and it expands the string and echoes it.
In the second case, the absence of quotes means that the first shell expands the information and it sees echo 3.2.57(1)-release, and if you type that at the command line, you get the syntax error.
If you used echo "$BASH_VERSION" in both, then both would work, but different shells would expand $BASH_VERSION.
There is a quite nasty expression that want to echo using bash.
The expression is:
'one two --
Note: There is white space after --.
So I have:
IFS=
echo 'one$IFStwoIFS--$IFS
But the result is:
one$IFStwo$IFS--$IFS
You have few issues with your approach:
Within single quote variables are not expanded in shell
In the string one$IFStwo$IFS--$IFS first instance of $IFS will not be expanded since you have string two next to $IFS so it attempts to expand non-existent variable $IFStwo.
Default value of $IFS is $' \t\n'
You can use:
echo "one${IFS}two$IFS--$IFS"
which will expand to (cat -A output):
one ^I$
two ^I$
-- ^I$
shell scripting features can be disabled in a here-document by escaping characters in the ending marker defined on the first line. For example:
$ cat <<'EOF'
→ $PATH
→ EOF
$PATH
Is there any way to do the same thing with a here-string so that one doesn't have to prepend each variable with a backslash?
$ cat <<< $PATH expands the variable before executing the bash command
Just backslash the dollar sign:
cat <<< \$PATH
or use single quotes
cat <<< '$PATH'
I am writing a BASH script and I am using the bash command. Which one of the following is correct (or are both incorrect)?
bash $pbs_dir/${module_name}.${target_ID}.${instance_ID}.pbs
or
bash \$pbs_dir/\${module_name}.\${target_ID}.\${instance_ID}.pbs
\$ will be expanded to literal $, so there is a big difference:
$ a="hello"
$ echo $a
hello
$ echo \$a
$a
Also note that you almost always want to double quote your parameter expansions to avoid word splitting and pathname expansion:
echo "$a"
So you properly want to use the following:
bash "$pbs_dir/${module_name}.${target_ID}.${instance_ID}.pbs"