I'm trying to understand how the bash works from the inside, and I have a problem understanding why the heredoc's output gets expanded when we set a delimiter without quotes and get expanded with them.
For example :
Delimiter without quotes
bash-3.2$ cat << a
> test
> $SHELL
> a
test
/bin/zsh
Delimiter with quotes :
bash-3.2$ cat << 'a'
> test
> $SHELL
> a
test
$SHELL
Quoting the heredoc delimiter as 'a' or "a" or \a or any other way causes the heredoc to be treated literally with expansion disabled. This is a useful feature for cases where you don't want dollar signs and other special characters to be expanded: for example, when the heredoc contains an embedded shell script.
This behavior is described in the Bash manual (emphasis added):
3.6.6 Here Documents
This type of redirection instructs the shell to read input from the current source until a line containing only word (with no trailing blanks) is seen. All of the lines read up to that point are then used as the standard input (or file descriptor n if n is specified) for a command.
The format of here-documents is:
[n]<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic expansion, or filename expansion is performed on word. If any part of word is quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion, the character sequence \newline is ignored, and \ must be used to quote the characters \, $, and `.
If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing delimiter. This allows here-documents within shell scripts to be indented in a natural fashion.
Related
Is it possible to create a heredoc that does not become subject to variable expansion?
e.g.
cat <<-EOF > somefile.sh
Do not print current value of $1 instead evaluate it later.
EOF
Update I am aware of escaping by \. My actual heredoc has many variables in it - and it is error prone and tedious to escape all of them.
Quote the delimiter:
cat <<-"EOF" > somefile.sh
Do not print current value of $1 instead evaluate it later.
EOF
This results in:
$ cat somefile.sh
Do not print current value of $1 instead evaluate it later.
Documentation
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word. If
any characters in word are quoted, the delimiter is the result of
quote removal on word, and the lines in the here-document are
not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command
substitution, and arithmetic expansion, the character sequence
\ is ignored, and \ must be used to quote the characters \,
$, and `.
If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing
delimiter. This allows here-documents within shell scripts to be
indented in a natural fashion. [Emphasis added.]
Put backlash before the $ sign
$ VAR=XXX
$ cat << END
> dk
> \$VAR
> END
dk
$VAR
I write the following shell script:
#!/bin/sh
TEST_VAR="HELLO"
cat > test-1.txt << EOF
TEST_VAR is ${TEST_VAR}
EOF
cat > test-2.txt << "EOF"
TEST_VAR is ${TEST_VAR}
EOF
But I found the results are different:
test-1.txt is TEST_VAR is HELLO
And:
test-2.txt is TEST_VAR is ${TEST_VAR}
Could anyone help to explain it?
The " in the following line is causing the issue:
cat > test-2.txt << "EOF"
The reason is in man bash: when quoted there is no parameter expansion.
Here Documents
This type of redirection instructs the shell to read input from the current source until a line containing only delimiter (with no trailing blanks) is seen. All of the lines read
up to that point are then used as the standard input for a command.
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word. If any characters in word are quoted, the delimiter is
the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion,
command substitution, and arithmetic expansion, the character sequence \<newline> is ignored, and \ must be used to quote the characters \, $, and `.
If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing delimiter. This allows here-documents within shell scripts
to be indented in a natural fashion.
"EOF" will automatically escape all special characters, and print them as is, where EOF does not, and therefore puts the value.
Instead of "EOF" you could also use \EOF.
Is it possible to create a heredoc that does not become subject to variable expansion?
e.g.
cat <<-EOF > somefile.sh
Do not print current value of $1 instead evaluate it later.
EOF
Update I am aware of escaping by \. My actual heredoc has many variables in it - and it is error prone and tedious to escape all of them.
Quote the delimiter:
cat <<-"EOF" > somefile.sh
Do not print current value of $1 instead evaluate it later.
EOF
This results in:
$ cat somefile.sh
Do not print current value of $1 instead evaluate it later.
Documentation
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word. If
any characters in word are quoted, the delimiter is the result of
quote removal on word, and the lines in the here-document are
not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command
substitution, and arithmetic expansion, the character sequence
\ is ignored, and \ must be used to quote the characters \,
$, and `.
If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing
delimiter. This allows here-documents within shell scripts to be
indented in a natural fashion. [Emphasis added.]
Put backlash before the $ sign
$ VAR=XXX
$ cat << END
> dk
> \$VAR
> END
dk
$VAR
I'm a newbie to shell scripting and I have the following problem:
If I enter into the shell
cat << EOF
'"$10^2$"'
EOF
I expected (and wanted) to get something like
"$10^2$"
but actually it used parameter substitution and I got
'"0^2"'
Why does it parameter substitution (I used single quotes!?)? And how can I get the desired output?
Escape the dollar sign:
cat << EOF
'"\$10^2$"'
EOF
OUTPUT:
"$10^2$"
Otherwise $1 is being expanded by shell and shows output as empty string only only and you get output as '"0^2$"'
UPDATE: Otherwise use this form of heredoc to avoid variable expansion:
cat <<'EOF'
'"$10^2$"'
EOF
'"$10^2$"'
Explanation: As per man bash:
<<[-]word
here-document
delimiter
No parameter expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word. If any characters in word are quoted, the
delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document
are subjected to parameter expansion, command substitution, and arithmetic expansion. In the latter case, the character sequence \ is ignored, and
\ must be used to quote the characters \, $, and `.
See bolded text above for the explanation why variables got expanded in your example but not when I used quoted 'EOF'.
I have a little problem with my bash script.
#!/bin/bash
ex xxx.html << "HERE"
1,$s/\(foo\)/$1\1/
wq
HERE
This is just a little piece of my script. when I run it this is the output.
$1foo
Any way to fix this so the $1 will be the argument given to the script?
Thanks!
Try replacing "HERE" with HERE (unquoted). Also 1,$s becomes 1,\$s.
Here Documents
This type of redirection instructs the shell to read input from the
current source until a line containing only delimiter (with no trailing
blanks) is seen. All of the lines read up to that point are then used
as the standard input for a command.
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter expansion, command substitution, arithmetic expansion, or
pathname expansion is performed on word. If any characters in word are
quoted, the delimiter is the result of quote removal on word, and the
lines in the here-document are not expanded. If word is unquoted, all
lines of the here-document are subjected to parameter expansion, com-
mand substitution, and arithmetic expansion. In the latter case, the
character sequence \<newline> is ignored, and \ must be used to quote
the characters \, $, and `.
If the redirection operator is <<-, then all leading tab characters are
stripped from input lines and the line containing delimiter. This
allows here-documents within shell scripts to be indented in a natural
fashion.
Bash manual.
Remplace "HERE" with HERE (without quotes) and remplace 1,$ with 1,\$ or %
You could write your script as follows:
#!/bin/bash
ex xxx.html <<-HERE
%s/foo/$1&/
x
HERE
Although you could also build a smaller script:
#!/bin/bash
sed -i "s/foo/$1&/g" xxx.html
Try this.
#!/bin/bash
(echo '1,$s/\(foo\)/'"$1"'\1/'; echo 'wq') | ex xxx.html
That's 1,$s/\(foo\)/ in single quotes, adjacent to $1 in double quotes (so the shell substitutes the parameter), adjacent to \1/ in single quotes.