I have the following code in a shell script.
This only seems to work when it is not defined in a function.
The problematic line is the one containing the "<<".
The error message is
"./run: line 210: syntax error:
unexpected end of file"
How can I write this correctly within a function?
init_database()
{
cd ../cfg
db.sh << ENDC
$DB_ADMIN
0
y
n
ENDC
check_status
sqlplus $DB_SCHEMA#$DB_NAME < initial_data.sql
cd -
}
There are a number of ways to fix that problem.
1/ Unindent the here document end marker, such as:
cat <<EOF
hello
$PWD
EOF
but that will make your code look ugly.
2/ "Indent" the here document begin marker:
cat <<' EOF'
hello
$PWD
EOF
where that bit before the first EOF is exactly the same as the before the second (tab, four spaces, two tabs, whatever). This allows you to keep your nice indenting, although it doesn't expand variables inside the here-document ($PWD doesn't change).
3/ Allow tabs to be stripped from the start of input lines and the end marker.
cat <<-EOF
hello
$PWD
EOF
but there's no way to get tabs into the beginnings of lines.
4/ For your purposes, you can also use:
( echo "$DB_ADMIN";
echo "" ;
echo "0" ;
echo "y" ;
echo "n"
) | db.sh
check_status
sqlplus $DB_SCHEMA#$DB_NAME < initial_data.sql
cd -
I believe number 4 is the best option for you. It allows nice lining up of the input, tabs and spaces anywhere in the lines and variable expansion.
The end of your "Here document" needs to be unindented, I'm afraid.
The ENDC label must be alone in a line without leading/trailing whitspaces.
Related
I have file (Name test.func) with a comments as below
#--------------------
# DOG $ CAT NAMES
#--------------------
Brownie
Blacky
Vicky
Pammy
#--------------
# MOBILE & LAPTOP NAMES
#--------------
Lenovo
Oppo
Realme
The code i have written is as below
TestFile=$(cat /usr/test.func)
for line in $TestFile
echo "line is $line"
if [[ "$line" == *"#"* ]]; then
echo "$line is commented"
echo "$line" >>test_copy.func
echo " "
fi
if ...
#Some other logic here
fi
done
Output is giving as below (in test_copy.func)
line is #----------
#-------- is commented
line is #
# is commented
line is DOG
line is &
line is CAT
line is NAMES
*Some logic is performed*
line is #----------
#-------- is commented
line is #
# is commented
line is MOBILE
line is &
line is LAPTOP
line is NAMES
*Some logic is performed*
Expected output in test_copy.func should be as below
#--------------------
# DOG $ CAT NAMES
#--------------------
*Output as per the logic*
#--------------
# MOBILE & LAPTOP NAMES
#--------------
*Output as per the logic*
Commented lines are splited in the actual output.
But Expected result should be as in the source file
Can anyone help me to resolve this issue
code
The code:
TestFile=$(cat /usr/test.func)
for line in $TestFile
does not loop over the lines of the file, but over the "words" (contiguous strings of non-whitespace characters). The variable TestFile contains the contents of the file, but the for loop is subject to field splitting. In other words, if the file contains "foo bar baz", the loop is equivalent to for line in foo bar baz; do .... This is a very fragile construction, as it is also subject to glob expansion, etc. For example, if the file contains wildcards (eg foo * bar), those wildcards will be expanded (and foo * bar expands to a string that contains all the names in the current directory).
The standard way to iterate over the lines of a file is
while read line; do ... done < /usr/test.func
But this is terribly slow and should generally be avoided. Tools like sed and awk are far more appropriate. It's normally a bad idea to read through a file on multiple passes, but while read is so slow that you could read the file 50 times with other tools before you would likely begin to notice. You probably don't want to copy lines that merely contain a # (as the *"#"* expression will do, but only want to copy lines that begin with #, but that's a different question). I would recommend either:
sed -n -e '/^\s*#/p' /usr/test.func > test_copy.func
while read -r line; do some_other_logic "$line"; done < /usr/test.func
or:
awk '/^\s*#/{print > "test_copy.func"}
{ some other logic here }' /usr/test.func
I have a markdown file that has snippets of code resembling the following example:
```
$ cat docs/code_sample.sh
#!/usr/bin/env bash
echo "Hello, world"
```
This means there there's a file at the location docs/code_sample.sh, whose contents is:
#!/usr/bin/env bash
echo "Hello, world"
I'd like to parse the markdown file with sed (awk or perl works too) and replace the bottom section of the code snippet with whatever the above bash command evaluates to, for example whatever cat docs/code_sample.sh evaluates to.
Perl to the rescue!
perl -0777 -pe 's/(?<=```\n)^(\$ (.*)\n\n)(?^s:.*?)(?=```)/"$1".qx($2)/meg' < input > output
-0777 slurps the whole file into memory
-p prints the input after processing
s/PATTERN/REPLACEMENT/ works similarly to a substitution in sed
/g replaces globally, i.e. as many times as it can
/m makes ^ match start of each line instead of start of the whole input string
/e evaluates the replacement as code
(?<=```\n) means "preceded by three backquotes and a newline"
(?^s:.*?) changes the behaviour of . to match newlines as well, so it matches (frugally because of the *?) the rest of the preformatted block
(?=```) means "followed by three backquotes`
qx runs the parameter in a shell and returns its output
A sed-only solution is easier if you have the GNU version with an e command.
That said, here's a quick, simplistic, and kinda clumsy version I knocked out that doesn't bother to check the values of previous or following lines - it just assumes your format is good, and bulls through without any looping or anything else. Still, for my example code, it worked.
I started by making an a, a b, and an x that is the markup file.
$: cat a
#! /bin/bash
echo "Hello, World!"
$: cat b
#! /bin/bash
echo "SCREW YOU!!!!"
$: cat x
```
$ cat a
foo
bar
" b a z ! "
```
```
$ cat b
foo
bar
" b a z ! "
```
Then I wrote s which is the sed script.
$: cat s
#! /bin/env bash
sed -En '
/^```$/,/^```$/ {
# for the lines starting with the $ prompt
/^[$] / {
# save the command to the hold space
x
# write the ``` header to the pattern space
s/.*/```/
# print the fabricated header
p
# swap the command back in
x
# the next line should be blank - add it to the current pattern space
N
# first print the line of code as-is with the (assumed) following blank line
p
# scrub the $ (prompt) off the command
s/^[$] //
# execute the command - store the output into the pattern space
e
# print the output
p
# put the markdown footer back
s/.*/```/
# and print that
p
}
# for the (to be discarded) existing lines of "content"
/^[^`$]/d
}
' $*
It does the job and might get you started.
$: s x
```
$ cat a
#! /bin/bash
echo "Hello, World!"
```
```
$ cat b
#! /bin/bash
echo "SCREW YOU!!!!"
```
Lots of caveats - better to actually check that the $ follows a line of backticks and is followed by a blank line, maybe make sure nothing bogus could be in the file to get executed... but this does what you asked, with (GNU) sed.
Good luck.
A rare case when use of getline would be appropriate:
$ cat tst.awk
state == "importing" {
while ( (getline line < $NF) > 0 ) {
print line
}
close($NF)
state = "imported"
}
$0 == "```" { state = (state ? "" : "importing") }
state != "imported" { print }
$ awk -f tst.awk file
See http://awk.freeshell.org/AllAboutGetline for getline uses and caveats.
I needed to write a script to enter multi-line input to a program (psql).
After a bit of googling, I found the following syntax works:
cat << EOF | psql ---params
BEGIN;
`pg_dump ----something`
update table .... statement ...;
END;
EOF
This correctly constructs the multi-line string (from BEGIN; to END;, inclusive) and pipes it as an input to psql.
But I have no idea how/why it works, can some one please explain?
I'm referring mainly to cat << EOF, I know > outputs to a file, >> appends to a file, < reads input from file.
What does << exactly do?
And is there a man page for it?
The cat <<EOF syntax is very useful when working with multi-line text in Bash, eg. when assigning multi-line string to a shell variable, file or a pipe.
Examples of cat <<EOF syntax usage in Bash:
1. Assign multi-line string to a shell variable
$ sql=$(cat <<EOF
SELECT foo, bar FROM db
WHERE foo='baz'
EOF
)
The $sql variable now holds the new-line characters too. You can verify with echo -e "$sql".
2. Pass multi-line string to a file in Bash
$ cat <<EOF > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
EOF
The print.sh file now contains:
#!/bin/bash
echo $PWD
echo /home/user
3. Pass multi-line string to a pipe in Bash
$ cat <<EOF | grep 'b' | tee b.txt
foo
bar
baz
EOF
The b.txt file contains bar and baz lines. The same output is printed to stdout.
This is called heredoc format to provide a string into stdin. See https://en.wikipedia.org/wiki/Here_document#Unix_shells for more details.
From man bash:
Here Documents
This type of redirection instructs the shell to read input from
the current source until a line
containing only word (with no trailing
blanks) is seen.
All of the lines read up to that point are then used as the
standard input for a command.
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter expansion, command substitution, arithmetic expansion, or
pathname expansion is performed on
word. If any characters in word are
quoted, the
delimiter is the result of quote removal on word, and the lines
in the here-document are not expanded.
If word is unquoted, all lines of the
here-document are subjected to parameter expansion, command
substitution, and arithmetic
expansion. In the latter case, the
character sequence \<newline> is
ignored, and \ must be used to quote the characters \, $, and `.
If the redirection operator is <<-, then all leading tab characters
are stripped from input lines and the
line containing delimiter. This
allows here-documents within shell scripts to be indented in a natural fashion.
In your case, "EOF" is known as a "Here Tag". Basically <<Here tells the shell that you are going to enter a multiline string until the "tag" Here. You can name this tag as you want, it's often EOF or STOP.
Some rules about the Here tags:
The tag can be any string, uppercase or lowercase, though most people use uppercase by convention.
The tag will not be considered as a Here tag if there are other words in that line. In this case, it will merely be considered part of the string. The tag should be by itself on a separate line, to be considered a tag.
The tag should have no leading or trailing spaces in that line to be considered a tag. Otherwise it will be considered as part of the string.
example:
$ cat >> test <<HERE
> Hello world HERE <-- Not by itself on a separate line -> not considered end of string
> This is a test
> HERE <-- Leading space, so not considered end of string
> and a new line
> HERE <-- Now we have the end of the string
POSIX 7
kennytm quoted man bash, but most of that is also POSIX 7: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04 :
The redirection operators "<<" and "<<-" both allow redirection of lines contained in a shell input file, known as a "here-document", to the input of a command.
The here-document shall be treated as a single word that begins after the next <newline> and continues until there is a line containing only the delimiter and a <newline>, with no <blank> characters in between. Then the next here-document starts, if there is one. The format is as follows:
[n]<<word
here-document
delimiter
where the optional n represents the file descriptor number. If the number is omitted, the here-document refers to standard input (file descriptor 0).
If any character in word is quoted, the delimiter shall be formed by performing quote removal on word, and the here-document lines shall not be expanded. Otherwise, the delimiter shall be the word itself.
If no characters in word are quoted, all lines of the here-document shall be expanded for parameter expansion, command substitution, and arithmetic expansion. In this case, the <backslash> in the input behaves as the <backslash> inside double-quotes (see Double-Quotes). However, the double-quote character ( '"' ) shall not be treated specially within a here-document, except when the double-quote appears within "$()", "``", or "${}".
If the redirection symbol is "<<-", all leading <tab> characters shall be stripped from input lines and the line containing the trailing delimiter. If more than one "<<" or "<<-" operator is specified on a line, the here-document associated with the first operator shall be supplied first by the application and shall be read first by the shell.
When a here-document is read from a terminal device and the shell is interactive, it shall write the contents of the variable PS2, processed as described in Shell Variables, to standard error before reading each line of input until the delimiter has been recognized.
Examples
Some examples not yet given.
Quotes prevent parameter expansion
Without quotes:
a=0
cat <<EOF
$a
EOF
Output:
0
With quotes:
a=0
cat <<'EOF'
$a
EOF
or (ugly but valid):
a=0
cat <<E"O"F
$a
EOF
Outputs:
$a
Hyphen removes leading tabs
Without hyphen:
cat <<EOF
<tab>a
EOF
where <tab> is a literal tab, and can be inserted with Ctrl + V <tab>
Output:
<tab>a
With hyphen:
cat <<-EOF
<tab>a
<tab>EOF
Output:
a
This exists of course so that you can indent your cat like the surrounding code, which is easier to read and maintain. E.g.:
if true; then
cat <<-EOF
a
EOF
fi
Unfortunately, this does not work for space characters: POSIX favored tab indentation here. Yikes.
Using tee instead of cat
Not exactly as an answer to the original question, but I wanted to share this anyway: I had the need to create a config file in a directory that required root rights.
The following does not work for that case:
$ sudo cat <<EOF >/etc/somedir/foo.conf
# my config file
foo=bar
EOF
because the redirection is handled outside of the sudo context.
I ended up using this instead:
$ sudo tee <<EOF /etc/somedir/foo.conf >/dev/null
# my config file
foo=bar
EOF
<< EoF basically means:
<< - "read the multi-line input that begins from the next line onward, and treat it as if it's code in a separate file"
EoF - "stop reading immediately after the word EoF is found in the multi-line input"
As other answers have explained, the multi-line input is called a Here Document
A Here Document is often used to generate output to be passed to a subsequent process. For example cat << EoF can be used to generate a desired output, using a Here Document.
Here's an example of using a Here Document to create a text document on the fly:
cat << EoF > ./my-document.txt
Hello world
Have a nice day
EoF
A little extension to the above answers. The trailing > directs the input into the file, overwriting existing content. However, one particularly convenient use is the double arrow >> that appends, adding your new content to the end of the file, as in:
cat <<EOF >> /etc/fstab
data_server:/var/sharedServer/authority/cert /var/sharedFolder/sometin/authority/cert nfs
data_server:/var/sharedServer/cert /var/sharedFolder/sometin/vsdc/cert nfs
EOF
This extends your fstab without you having to worry about accidentally modifying any of its contents.
note to mention that cat << \EOT (see the backslash) will not expand any variables inside, while cat << EOT will do.
examples:
FOO="bar"
cat << \EOT > foobar.txt
echo "$FOO"
EOT
will output:
echo $FOO
while:
FOO="bar"
cat << EOT > foobar.txt
echo "$FOO"
EOT
will output:
echo "bar"
Example to create a json file:
cat << EoF > ./allaccess.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
EoF
As a result:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
Long story short, EOF marker(but a different literal can be used as well) is a heredoc format that allows you to provide your input as multiline.
A lot of confusion comes from how cat actually works it seems.
You can use cat with >> or > as follows:
$ cat >> temp.txt
line 1
line 2
While cat can be used this way when writing manually into console, it's not convenient if I want to provide the input in a more declarative way so that it can be reused by tools and also to keep indentations, whitespaces, etc.
Heredoc allows to define your entire input as if you are not working with stdin but typing in a separate text editor. This is what Wikipedia article means by:
it is a section of a source code file that is treated as if it were a
separate file.
This isn't necessarily an answer to the original question, but a sharing of some results from my own testing. This:
<<test > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
test
will produce the same file as:
cat <<test > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
test
So, I don't see the point of using the cat command.
Worth noting that here docs work in bash loops too.
This example shows how-to get the column list of table:
export postgres_db_name='my_db'
export table_name='my_table_name'
# start copy
while read -r c; do test -z "$c" || echo $table_name.$c , ; done < <(cat << EOF | psql -t -q -d $postgres_db_name -v table_name="${table_name:-}"
SELECT column_name
FROM information_schema.columns
WHERE 1=1
AND table_schema = 'public'
AND table_name =:'table_name' ;
EOF
)
# stop copy , now paste straight into the bash shell ...
output:
my_table_name.guid ,
my_table_name.id ,
my_table_name.level ,
my_table_name.seq ,
or even without the new line
while read -r c; do test -z "$c" || echo $table_name.$c , | perl -ne
's/\n//gm;print' ; done < <(cat << EOF | psql -t -q -d $postgres_db_name -v table_name="${table_name:-}"
SELECT column_name
FROM information_schema.columns
WHERE 1=1
AND table_schema = 'public'
AND table_name =:'table_name' ;
EOF
)
# output: daily_issues.guid ,daily_issues.id ,daily_issues.level ,daily_issues.seq ,daily_issues.prio ,daily_issues.weight ,daily_issues.status ,daily_issues.category ,daily_issues.name ,daily_issues.description ,daily_issues.type ,daily_issues.owner
inside of a function, I would have had multiple echos, but I remembered I could do cat << EOF followed by my text to output, followed by EOF. However, it only seems to work when the EOF part is not indented, like this
init(){
if [ conditionial ]; then
cat << EOF
this is some text
this is more text
this is even more text
EOF
but it doesn't work like this:
init(){
if [ conditionial ]; then
cat << EOF
this is some text
this is more text
this is even more text
EOF
Any ideas?
"Here document" terminators must appear at the beginning of the line, unless the redirection operator has a - suffix:
cat << END
this does not
END
here, but rather here
END
vs:
cat <<- END
this one does end here
END
(watch out for spaces vs tabs; and some older shell variants do not support this).
Besides these, another useful trick is the handling of expansions inside here documents. If the end word (END in my examples above, EOF in yours) is quoted, expansion is inhibited:
cat << E1
you are in a maze of twisty little directories: $PWD
E1
cat << 'E2'
you owe the Usenet Oracle $1.75 for the Tab.
E2
I never knew this till you asked, but you can use <<- (note the -) to ignore leading tabs (not spaces).
See here
The EOF has to in the beginning of the line and It cant't have anything after it but end of line:
#! /bin/bash
function say_hello {
cat << EOF
hello
world
EOF
}
Execution:
bash test.sh
hello
world
Taking a closer look with cat -A:
cat -A test.sh
#! /bin/bash$
$
function say_hello {$
^Icat << EOF$
^Ihello$
^Iworld$
EOF$
}$
$
say_hello$
I had problem with script which would have space after EOF and would not work, only with cat -A I could identify the space.
I've got a script 'myscript' that outputs the following:
abc
def
ghi
in another script, I call:
declare RESULT=$(./myscript)
and $RESULT gets the value
abc def ghi
Is there a way to store the result either with the newlines, or with '\n' character so I can output it with 'echo -e'?
Actually, RESULT contains what you want — to demonstrate:
echo "$RESULT"
What you show is what you get from:
echo $RESULT
As noted in the comments, the difference is that (1) the double-quoted version of the variable (echo "$RESULT") preserves internal spacing of the value exactly as it is represented in the variable — newlines, tabs, multiple blanks and all — whereas (2) the unquoted version (echo $RESULT) replaces each sequence of one or more blanks, tabs and newlines with a single space. Thus (1) preserves the shape of the input variable, whereas (2) creates a potentially very long single line of output with 'words' separated by single spaces (where a 'word' is a sequence of non-whitespace characters; there needn't be any alphanumerics in any of the words).
Another pitfall with this is that command substitution — $() — strips trailing newlines. Probably not always important, but if you really want to preserve exactly what was output, you'll have to use another line and some quoting:
RESULTX="$(./myscript; echo x)"
RESULT="${RESULTX%x}"
This is especially important if you want to handle all possible filenames (to avoid undefined behavior like operating on the wrong file).
In case that you're interested in specific lines, use a result-array:
declare RESULT=($(./myscript)) # (..) = array
echo "First line: ${RESULT[0]}"
echo "Second line: ${RESULT[1]}"
echo "N-th line: ${RESULT[N]}"
In addition to the answer given by #l0b0 I just had the situation where I needed to both keep any trailing newlines output by the script and check the script's return code.
And the problem with l0b0's answer is that the 'echo x' was resetting $? back to zero... so I managed to come up with this very cunning solution:
RESULTX="$(./myscript; echo x$?)"
RETURNCODE=${RESULTX##*x}
RESULT="${RESULTX%x*}"
Parsing multiple output
Introduction
So your myscript output 3 lines, could look like:
myscript() { echo $'abc\ndef\nghi'; }
or
myscript() { local i; for i in abc def ghi ;do echo $i; done ;}
Ok this is a function, not a script (no need of path ./), but output is same
myscript
abc
def
ghi
Considering result code
To check for result code, test function will become:
myscript() { local i;for i in abc def ghi ;do echo $i;done;return $((RANDOM%128));}
1. Storing multiple output in one single variable, showing newlines
Your operation is correct:
RESULT=$(myscript)
About result code, you could add:
RCODE=$?
even in same line:
RESULT=$(myscript) RCODE=$?
Then
echo $RESULT $RCODE
abc def ghi 66
echo "$RESULT"
abc
def
ghi
echo ${RESULT#Q}
$'abc\ndef\nghi'
printf '%q\n' "$RESULT"
$'abc\ndef\nghi'
but for showing variable definition, use declare -p:
declare -p RESULT RCODE
declare -- RESULT="abc
def
ghi"
declare -- RCODE="66"
2. Parsing multiple output in array, using mapfile
Storing answer into myvar variable:
mapfile -t myvar < <(myscript)
echo ${myvar[2]}
ghi
Showing $myvar:
declare -p myvar
declare -a myvar=([0]="abc" [1]="def" [2]="ghi")
Considering result code
In case you have to check for result code, you could:
RESULT=$(myscript) RCODE=$?
mapfile -t myvar <<<"$RESULT"
declare -p myvar RCODE
declare -a myvar=([0]="abc" [1]="def" [2]="ghi")
declare -- RCODE="40"
3. Parsing multiple output by consecutives read in command group
{ read firstline; read secondline; read thirdline;} < <(myscript)
echo $secondline
def
Showing variables:
declare -p firstline secondline thirdline
declare -- firstline="abc"
declare -- secondline="def"
declare -- thirdline="ghi"
I often use:
{ read foo;read foo total use free foo ;} < <(df -k /)
Then
declare -p use free total
declare -- use="843476"
declare -- free="582128"
declare -- total="1515376"
Considering result code
Same prepended step:
RESULT=$(myscript) RCODE=$?
{ read firstline; read secondline; read thirdline;} <<<"$RESULT"
declare -p firstline secondline thirdline RCODE
declare -- firstline="abc"
declare -- secondline="def"
declare -- thirdline="ghi"
declare -- RCODE="50"
After trying most of the solutions here, the easiest thing I found was the obvious - using a temp file. I'm not sure what you want to do with your multiple line output, but you can then deal with it line by line using read. About the only thing you can't really do is easily stick it all in the same variable, but for most practical purposes this is way easier to deal with.
./myscript.sh > /tmp/foo
while read line ; do
echo 'whatever you want to do with $line'
done < /tmp/foo
Quick hack to make it do the requested action:
result=""
./myscript.sh > /tmp/foo
while read line ; do
result="$result$line\n"
done < /tmp/foo
echo -e $result
Note this adds an extra line. If you work on it you can code around it, I'm just too lazy.
EDIT: While this case works perfectly well, people reading this should be aware that you can easily squash your stdin inside the while loop, thus giving you a script that will run one line, clear stdin, and exit. Like ssh will do that I think? I just saw it recently, other code examples here: https://unix.stackexchange.com/questions/24260/reading-lines-from-a-file-with-bash-for-vs-while
One more time! This time with a different filehandle (stdin, stdout, stderr are 0-2, so we can use &3 or higher in bash).
result=""
./test>/tmp/foo
while read line <&3; do
result="$result$line\n"
done 3</tmp/foo
echo -e $result
you can also use mktemp, but this is just a quick code example. Usage for mktemp looks like:
filenamevar=`mktemp /tmp/tempXXXXXX`
./test > $filenamevar
Then use $filenamevar like you would the actual name of a file. Probably doesn't need to be explained here but someone complained in the comments.
How about this, it will read each line to a variable and that can be used subsequently !
say myscript output is redirected to a file called myscript_output
awk '{while ( (getline var < "myscript_output") >0){print var;} close ("myscript_output");}'