Input syntax for heredoc in Bash - bash

I'm currently learning Unix bash, through the terminal in Ubuntu 16. I was writing a simple script, so I can practice writing code for Unix. Here is a sample of the code:
report_uptime(){
cat << _EOF_
<H2>System Uptime</H2>
<PRE>$(uptime)</PRE>
_EOF_
return
}
This code doesn't work and the reason is because after cat I'm supposed to use <<- instead of <<. Sometime << works. So when am I supposed to use << and when am I supposed to use<<-?
This works:
report_uptime(){
cat <<- _EOF_
<H2>System Uptime</H2>
<PRE>$(uptime)</PRE>
_EOF_
return
}

This isn't cat syntax per se; it's one of the redirection operators supported by your shell:
https://www.gnu.org/software/bash/manual/bashref.html#Here-Documents
3.6.6 Here Documents
This type of redirection instructs the shell to read input from the current source until a line containing only word (with no trailing blanks) is seen. All of the lines read up to that point are then used as the standard input for a command.
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic expansion, or filename expansion is performed on word. If any characters in word are quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion, the character sequence \newline is ignored, and ‘\’ must be used to quote the characters ‘\’, ‘$’, and ‘`’.
If the redirection operator is ‘<<-’, then all leading tab characters are stripped from input lines and the line containing delimiter. This allows here-documents within shell scripts to be indented in a natural fashion.

This is not strictly speaking about the syntax of the cat utility, but of the syntax of the shell itself, i.e. of bash.
The construct that you're using is called a "here-document". A here-document feeds its contents into the standard input of whatever command you put before <<.
The syntax is this:
command <<word
...
contents
...
END_TAG
The word here is either exactly the same as the END_TAG, or it may be 'END_TAG', -END_TAG, or -'END_TAG'.
END_TAG: Without single quotes, the contents of the here-document will undergo substitution. This means that any variable, or simply put, "anything that contains a $" will be replaced with its value.
$ tr 'a-z' 'A-Z' <<TR_END
> This is my $HOME
> TR_END
THIS IS MY /USERS/KK
(The >, a greater-than sign and a space, is what's called the secondary prompt. I get that because I'm typing this directly into the shell, and it needs more lines of input before it can execute the whole command. It is not typed by me.)
'END_TAG': With single quotes, the contents of the here-document will not undergo substitution. This means, for example, that if you write $HOME in the here-document, it will be fed into the command just like that, not like /home/myname (or whatever your home directory may be).
$ tr 'a-z' 'A-Z' <<'TR_END'
> This is my $HOME
> TR_END
THIS IS MY $HOME
With a leading dash (-), the shell will strip off all tabs (but not spaces) at the start of each line of the here-document, including the line with the END_TAG at the end.
$ tr 'a-z' 'A-Z' <<-TR_END
> This line has a tab.
> This one does not.
> TR_END
THIS LINE HAS A TAB.
THIS ONE DOES NOT.
Without a leading dash, the shell will not strip off tabs. The END_TAG needs to be the first (and only) thing on the line ending the here-document.
$ tr 'a-z' 'A-Z' <<TR_END
> This line has a tab.
> This one does not.
> TR_END
THIS LINE HAS A TAB.
THIS ONE DOES NOT.
The bash shell also has something called "here-strings". It works in a similar fashion, but you're only feeding a single line into the command:
command <<<word
For example:
$ tr 'a-z' 'A-Z' <<<"hello world!"
HELLO WORLD!

Generally, the end marker should not be indented. Move it to column 1. Also remove the return statement, and it works for me in bash.

Related

When is a heredoc's output expanded?

I'm trying to understand how the bash works from the inside, and I have a problem understanding why the heredoc's output gets expanded when we set a delimiter without quotes and get expanded with them.
For example :
Delimiter without quotes
bash-3.2$ cat << a
> test
> $SHELL
> a
test
/bin/zsh
Delimiter with quotes :
bash-3.2$ cat << 'a'
> test
> $SHELL
> a
test
$SHELL
Quoting the heredoc delimiter as 'a' or "a" or \a or any other way causes the heredoc to be treated literally with expansion disabled. This is a useful feature for cases where you don't want dollar signs and other special characters to be expanded: for example, when the heredoc contains an embedded shell script.
This behavior is described in the Bash manual (emphasis added):
3.6.6 Here Documents
This type of redirection instructs the shell to read input from the current source until a line containing only word (with no trailing blanks) is seen. All of the lines read up to that point are then used as the standard input (or file descriptor n if n is specified) for a command.
The format of here-documents is:
[n]<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic expansion, or filename expansion is performed on word. If any part of word is quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion, the character sequence \newline is ignored, and \ must be used to quote the characters \, $, and `.
If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing delimiter. This allows here-documents within shell scripts to be indented in a natural fashion.

How to use cat and EOF in a bash file [duplicate]

Is it possible to create a heredoc that does not become subject to variable expansion?
e.g.
cat <<-EOF > somefile.sh
Do not print current value of $1 instead evaluate it later.
EOF
Update I am aware of escaping by \. My actual heredoc has many variables in it - and it is error prone and tedious to escape all of them.
Quote the delimiter:
cat <<-"EOF" > somefile.sh
Do not print current value of $1 instead evaluate it later.
EOF
This results in:
$ cat somefile.sh
Do not print current value of $1 instead evaluate it later.
Documentation
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word. If
any characters in word are quoted, the delimiter is the result of
quote removal on word, and the lines in the here-document are
not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command
substitution, and arithmetic expansion, the character sequence
\ is ignored, and \ must be used to quote the characters \,
$, and `.
If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing
delimiter. This allows here-documents within shell scripts to be
indented in a natural fashion. [Emphasis added.]
Put backlash before the $ sign
$ VAR=XXX
$ cat << END
> dk
> \$VAR
> END
dk
$VAR

cat > somefile << "EOF" or EOF, that make differnet result on bash shell

I write the following shell script:
#!/bin/sh
TEST_VAR="HELLO"
cat > test-1.txt << EOF
TEST_VAR is ${TEST_VAR}
EOF
cat > test-2.txt << "EOF"
TEST_VAR is ${TEST_VAR}
EOF
But I found the results are different:
test-1.txt is TEST_VAR is HELLO
And:
test-2.txt is TEST_VAR is ${TEST_VAR}
Could anyone help to explain it?
The " in the following line is causing the issue:
cat > test-2.txt << "EOF"
The reason is in man bash: when quoted there is no parameter expansion.
Here Documents
This type of redirection instructs the shell to read input from the current source until a line containing only delimiter (with no trailing blanks) is seen. All of the lines read
up to that point are then used as the standard input for a command.
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter and variable expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word. If any characters in word are quoted, the delimiter is
the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion,
command substitution, and arithmetic expansion, the character sequence \<newline> is ignored, and \ must be used to quote the characters \, $, and `.
If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing delimiter. This allows here-documents within shell scripts
to be indented in a natural fashion.
"EOF" will automatically escape all special characters, and print them as is, where EOF does not, and therefore puts the value.
Instead of "EOF" you could also use \EOF.

Shell script multi-line comment

I am having a large shell script file. At times while doing modification I want to comment out part of it. But commenting line as shown in the below example is giving me error.
Script:
#!/bin/bash
<<COMMENT1
read build_label
read build_branch_tag
build_date_tag=$(echo $build_label | sed "s/$build_branch_tag//g")
echo $build_path
COMMENT1
echo "HELLO WORLD"
Error Message:
sed: first RE may not be empty
I just want to understand what's wrong with the above script and why comment section is not working properly.
First, using here docs to comment code is really dirty! Use the # instead. If you want to comment multiple lines, use your editor. In vim (commenting lines from 10 to 15 for example):
:10,15s/^/#
However, to solve your current problem you need to enclose the starting here-doc delimiter in single quotes, like this:
<<'COMMENT'
...
COMMENT
Using single quotes you tell bash that it should not attempt to expand variables or expression inside the here doc body.
Traditional UNIX shell doesn't have multiline comment support. What you're doing here is using a so-called "HERE document" without using its value, a common hack to get multiline comment like behaviour.
However, patterns inside the the HERE document are still evaluated, which means that your $(…) is executed. But since build_branch_tag has not been defined before, it will evaluate to an empty string, and the shell will thus execute sed s///g.
You can use a different hack:
: '
Bla bla, no $expansion is taking place here.
'
What this is doing: the : is a no-op command, it simply does nothing. And you're passing it an argument which is a string '…'. Inside the single quotes, no expansion/evaluation is taking place. Beware of ' inside the "commented out" region, though.
You can turn parameter substitution off inside a here document like this:
<<"Endofmessage"
or
<<'Endofmessage'
or
<<\Endofmessage
Here Documents
This type of redirection instructs the shell to read input from the
current source until a line containing only delimiter (with no
trailing blanks) is seen. All of the lines read up to that point are
then used as the standard input for a command. The format of
here-documents is:
<<[-]word
here-document delimiter No parameter expansion, command substitution, arithmetic expansion, or pathname expansion is performed
on word. If any characters in word are quoted, the delimiter is the
result of quote removal on word, and the lines in the here-document
are not expanded. If word is unquoted, all lines of the here-document
are subjected to parameter expansion, command substitution, and
arithmetic expansion. In the latter case, the character sequence
\ is ignored, and \ must be used to quote the characters \,
$, and `. If the redirection operator is <<-, then all leading tab
characters are stripped from input lines and the line containing
delimiter. This allows here-documents within shell scripts to be
indented in a natural fashion.
And maybe something that you may also like: I prefer to do multiline comments in my bash script with the nodepad++ shortcut ctrl+Q (toggle comment).
if this is not a syntax error (open string, ...)
#!/bin/bash
if false;then
read build_label
read build_branch_tag
build_date_tag=$(echo $build_label | sed "s/$build_branch_tag//g")
echo $build_path
fi
echo "HELLO WORLD"
if sysntax error or equivalent (unfound place like in search of error by descativate part of failing code)
#!/bin/bash
#read build_label
#read build_branch_tag
#build_date_tag=$(echo $build_label | sed "s/$build_branch_tag//g")
#echo $build_path
echo "HELLO WORLD"
for this you can use:
- editor if find/replace with regex is available like vi(m)
- a sed (sed '14,45 s/^/#/' YourFile > YourFile.Debug where 14 and 45 are first and last lines to comment)
Using here docs to comment code is safe and elegant like this:
: <<'EOT'
Example usage of the null command ':' and the here-document syntax for a
multi-line comment. If the delimiter word ('EOT' here) is quoted, the
here-document will not be expanded in any way. This is important, as
an unquoted delimiter will result in problems with unintended potential
expansions. All of this here-doc text is redirected to the standard input
of :, which does nothing but return true.
EOT

bash variable isn't working with ex in script

I have a little problem with my bash script.
#!/bin/bash
ex xxx.html << "HERE"
1,$s/\(foo\)/$1\1/
wq
HERE
This is just a little piece of my script. when I run it this is the output.
$1foo
Any way to fix this so the $1 will be the argument given to the script?
Thanks!
Try replacing "HERE" with HERE (unquoted). Also 1,$s becomes 1,\$s.
Here Documents
This type of redirection instructs the shell to read input from the
current source until a line containing only delimiter (with no trailing
blanks) is seen. All of the lines read up to that point are then used
as the standard input for a command.
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter expansion, command substitution, arithmetic expansion, or
pathname expansion is performed on word. If any characters in word are
quoted, the delimiter is the result of quote removal on word, and the
lines in the here-document are not expanded. If word is unquoted, all
lines of the here-document are subjected to parameter expansion, com-
mand substitution, and arithmetic expansion. In the latter case, the
character sequence \<newline> is ignored, and \ must be used to quote
the characters \, $, and `.
If the redirection operator is <<-, then all leading tab characters are
stripped from input lines and the line containing delimiter. This
allows here-documents within shell scripts to be indented in a natural
fashion.
Bash manual.
Remplace "HERE" with HERE (without quotes) and remplace 1,$ with 1,\$ or %
You could write your script as follows:
#!/bin/bash
ex xxx.html <<-HERE
%s/foo/$1&/
x
HERE
Although you could also build a smaller script:
#!/bin/bash
sed -i "s/foo/$1&/g" xxx.html
Try this.
#!/bin/bash
(echo '1,$s/\(foo\)/'"$1"'\1/'; echo 'wq') | ex xxx.html
That's 1,$s/\(foo\)/ in single quotes, adjacent to $1 in double quotes (so the shell substitutes the parameter), adjacent to \1/ in single quotes.

Resources