defineColumns() {
shift
local dirfun=${1:-"/var/log/was/dial"}
local basefun=${2:-"$logdir/party_info.$(date +%y%m%d%H%M%S)"}
touch $logbase_bcdb
info $dirfun $basefun
# Run steps sequentially
loadData
}
I am writing a shell script code in which I have written multiple functions. The above function is throwing error as :
syntax error near unexpected token `{
What is wrong in the code?
There are non-printing characters in the code: Unicode U+200B ZERO WIDTH SPACE. Remove them and you should be fine.
Firstly to see them, you could use cat -A but it shows these characters as M-bM-^#M-^K, which is confusing IMO. I'd rather read the Python ascii() representation, so here's a quick script:
import fileinput
for line in fileinput.input():
print(ascii(line))
Save that as ascii_lines.py then run with the name of your script:
$ python3 ascii_lines.py filename.sh
'\u200b\u200b\u200b\u200b\u200bdefineColumns() {\n'
'\u200b\u200b\u200b\u200b\u200b\u200b\u200b shift\n'
' local dirfun=${\u200b\u200b\u200b\u200b\u200b\u200b\u200b\u200b1:-"/var/log/was/dial"}\u200b\u200b\u200b\u200b\u200b\u200b\u200b\u200b\n'
' local basefun=${\u200b\u200b\u200b\u200b\u200b\u200b\u200b\u200b2:-"$logdir/party_info.$(date +%y%m%d%H%M%S)"}\u200b\u200b\u200b\u200b\u200b\u200b\u200b\u200b\n'
' touch $logbase_bcdb\n'
' info $dirfun $basefun\n'
' # Run steps sequentially\n'
' loadData\n'
'}\n'
Then to remove them, you could use sed, though it doesn't know Unicode, so I'm using a Bash $'' string here to solve that.
$ sed -i $'s/\u200b//g' filename.sh
Afterwards:
$ python3 ascii_lines.py filename
'defineColumns() {\n'
' shift\n'
' local dirfun=${1:-"/var/log/was/dial"}\n'
' local basefun=${2:-"$logdir/party_info.$(date +%y%m%d%H%M%S)"}\n'
' touch $logbase_bcdb\n'
' info $dirfun $basefun\n'
' # Run steps sequentially\n'
' loadData\n'
'}\n'
I needed to write a script to enter multi-line input to a program (psql).
After a bit of googling, I found the following syntax works:
cat << EOF | psql ---params
BEGIN;
`pg_dump ----something`
update table .... statement ...;
END;
EOF
This correctly constructs the multi-line string (from BEGIN; to END;, inclusive) and pipes it as an input to psql.
But I have no idea how/why it works, can some one please explain?
I'm referring mainly to cat << EOF, I know > outputs to a file, >> appends to a file, < reads input from file.
What does << exactly do?
And is there a man page for it?
The cat <<EOF syntax is very useful when working with multi-line text in Bash, eg. when assigning multi-line string to a shell variable, file or a pipe.
Examples of cat <<EOF syntax usage in Bash:
1. Assign multi-line string to a shell variable
$ sql=$(cat <<EOF
SELECT foo, bar FROM db
WHERE foo='baz'
EOF
)
The $sql variable now holds the new-line characters too. You can verify with echo -e "$sql".
2. Pass multi-line string to a file in Bash
$ cat <<EOF > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
EOF
The print.sh file now contains:
#!/bin/bash
echo $PWD
echo /home/user
3. Pass multi-line string to a pipe in Bash
$ cat <<EOF | grep 'b' | tee b.txt
foo
bar
baz
EOF
The b.txt file contains bar and baz lines. The same output is printed to stdout.
This is called heredoc format to provide a string into stdin. See https://en.wikipedia.org/wiki/Here_document#Unix_shells for more details.
From man bash:
Here Documents
This type of redirection instructs the shell to read input from
the current source until a line
containing only word (with no trailing
blanks) is seen.
All of the lines read up to that point are then used as the
standard input for a command.
The format of here-documents is:
<<[-]word
here-document
delimiter
No parameter expansion, command substitution, arithmetic expansion, or
pathname expansion is performed on
word. If any characters in word are
quoted, the
delimiter is the result of quote removal on word, and the lines
in the here-document are not expanded.
If word is unquoted, all lines of the
here-document are subjected to parameter expansion, command
substitution, and arithmetic
expansion. In the latter case, the
character sequence \<newline> is
ignored, and \ must be used to quote the characters \, $, and `.
If the redirection operator is <<-, then all leading tab characters
are stripped from input lines and the
line containing delimiter. This
allows here-documents within shell scripts to be indented in a natural fashion.
In your case, "EOF" is known as a "Here Tag". Basically <<Here tells the shell that you are going to enter a multiline string until the "tag" Here. You can name this tag as you want, it's often EOF or STOP.
Some rules about the Here tags:
The tag can be any string, uppercase or lowercase, though most people use uppercase by convention.
The tag will not be considered as a Here tag if there are other words in that line. In this case, it will merely be considered part of the string. The tag should be by itself on a separate line, to be considered a tag.
The tag should have no leading or trailing spaces in that line to be considered a tag. Otherwise it will be considered as part of the string.
example:
$ cat >> test <<HERE
> Hello world HERE <-- Not by itself on a separate line -> not considered end of string
> This is a test
> HERE <-- Leading space, so not considered end of string
> and a new line
> HERE <-- Now we have the end of the string
POSIX 7
kennytm quoted man bash, but most of that is also POSIX 7: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04 :
The redirection operators "<<" and "<<-" both allow redirection of lines contained in a shell input file, known as a "here-document", to the input of a command.
The here-document shall be treated as a single word that begins after the next <newline> and continues until there is a line containing only the delimiter and a <newline>, with no <blank> characters in between. Then the next here-document starts, if there is one. The format is as follows:
[n]<<word
here-document
delimiter
where the optional n represents the file descriptor number. If the number is omitted, the here-document refers to standard input (file descriptor 0).
If any character in word is quoted, the delimiter shall be formed by performing quote removal on word, and the here-document lines shall not be expanded. Otherwise, the delimiter shall be the word itself.
If no characters in word are quoted, all lines of the here-document shall be expanded for parameter expansion, command substitution, and arithmetic expansion. In this case, the <backslash> in the input behaves as the <backslash> inside double-quotes (see Double-Quotes). However, the double-quote character ( '"' ) shall not be treated specially within a here-document, except when the double-quote appears within "$()", "``", or "${}".
If the redirection symbol is "<<-", all leading <tab> characters shall be stripped from input lines and the line containing the trailing delimiter. If more than one "<<" or "<<-" operator is specified on a line, the here-document associated with the first operator shall be supplied first by the application and shall be read first by the shell.
When a here-document is read from a terminal device and the shell is interactive, it shall write the contents of the variable PS2, processed as described in Shell Variables, to standard error before reading each line of input until the delimiter has been recognized.
Examples
Some examples not yet given.
Quotes prevent parameter expansion
Without quotes:
a=0
cat <<EOF
$a
EOF
Output:
0
With quotes:
a=0
cat <<'EOF'
$a
EOF
or (ugly but valid):
a=0
cat <<E"O"F
$a
EOF
Outputs:
$a
Hyphen removes leading tabs
Without hyphen:
cat <<EOF
<tab>a
EOF
where <tab> is a literal tab, and can be inserted with Ctrl + V <tab>
Output:
<tab>a
With hyphen:
cat <<-EOF
<tab>a
<tab>EOF
Output:
a
This exists of course so that you can indent your cat like the surrounding code, which is easier to read and maintain. E.g.:
if true; then
cat <<-EOF
a
EOF
fi
Unfortunately, this does not work for space characters: POSIX favored tab indentation here. Yikes.
Using tee instead of cat
Not exactly as an answer to the original question, but I wanted to share this anyway: I had the need to create a config file in a directory that required root rights.
The following does not work for that case:
$ sudo cat <<EOF >/etc/somedir/foo.conf
# my config file
foo=bar
EOF
because the redirection is handled outside of the sudo context.
I ended up using this instead:
$ sudo tee <<EOF /etc/somedir/foo.conf >/dev/null
# my config file
foo=bar
EOF
<< EoF basically means:
<< - "read the multi-line input that begins from the next line onward, and treat it as if it's code in a separate file"
EoF - "stop reading immediately after the word EoF is found in the multi-line input"
As other answers have explained, the multi-line input is called a Here Document
A Here Document is often used to generate output to be passed to a subsequent process. For example cat << EoF can be used to generate a desired output, using a Here Document.
Here's an example of using a Here Document to create a text document on the fly:
cat << EoF > ./my-document.txt
Hello world
Have a nice day
EoF
A little extension to the above answers. The trailing > directs the input into the file, overwriting existing content. However, one particularly convenient use is the double arrow >> that appends, adding your new content to the end of the file, as in:
cat <<EOF >> /etc/fstab
data_server:/var/sharedServer/authority/cert /var/sharedFolder/sometin/authority/cert nfs
data_server:/var/sharedServer/cert /var/sharedFolder/sometin/vsdc/cert nfs
EOF
This extends your fstab without you having to worry about accidentally modifying any of its contents.
note to mention that cat << \EOT (see the backslash) will not expand any variables inside, while cat << EOT will do.
examples:
FOO="bar"
cat << \EOT > foobar.txt
echo "$FOO"
EOT
will output:
echo $FOO
while:
FOO="bar"
cat << EOT > foobar.txt
echo "$FOO"
EOT
will output:
echo "bar"
Example to create a json file:
cat << EoF > ./allaccess.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
EoF
As a result:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
Long story short, EOF marker(but a different literal can be used as well) is a heredoc format that allows you to provide your input as multiline.
A lot of confusion comes from how cat actually works it seems.
You can use cat with >> or > as follows:
$ cat >> temp.txt
line 1
line 2
While cat can be used this way when writing manually into console, it's not convenient if I want to provide the input in a more declarative way so that it can be reused by tools and also to keep indentations, whitespaces, etc.
Heredoc allows to define your entire input as if you are not working with stdin but typing in a separate text editor. This is what Wikipedia article means by:
it is a section of a source code file that is treated as if it were a
separate file.
This isn't necessarily an answer to the original question, but a sharing of some results from my own testing. This:
<<test > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
test
will produce the same file as:
cat <<test > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
test
So, I don't see the point of using the cat command.
Worth noting that here docs work in bash loops too.
This example shows how-to get the column list of table:
export postgres_db_name='my_db'
export table_name='my_table_name'
# start copy
while read -r c; do test -z "$c" || echo $table_name.$c , ; done < <(cat << EOF | psql -t -q -d $postgres_db_name -v table_name="${table_name:-}"
SELECT column_name
FROM information_schema.columns
WHERE 1=1
AND table_schema = 'public'
AND table_name =:'table_name' ;
EOF
)
# stop copy , now paste straight into the bash shell ...
output:
my_table_name.guid ,
my_table_name.id ,
my_table_name.level ,
my_table_name.seq ,
or even without the new line
while read -r c; do test -z "$c" || echo $table_name.$c , | perl -ne
's/\n//gm;print' ; done < <(cat << EOF | psql -t -q -d $postgres_db_name -v table_name="${table_name:-}"
SELECT column_name
FROM information_schema.columns
WHERE 1=1
AND table_schema = 'public'
AND table_name =:'table_name' ;
EOF
)
# output: daily_issues.guid ,daily_issues.id ,daily_issues.level ,daily_issues.seq ,daily_issues.prio ,daily_issues.weight ,daily_issues.status ,daily_issues.category ,daily_issues.name ,daily_issues.description ,daily_issues.type ,daily_issues.owner
This question already has answers here:
here-document gives 'unexpected end of file' error
(10 answers)
Closed 6 years ago.
I have the following function;
function getdetails {
if ! "${PSQL_PATH}" -d "$DB" -U "$DB_USER" -h localhost -p "$DB_PORT" -t -c | while read -a Record ; do
taskid="${Record[0]}"
clientname="${Record[1]}"
backup_pass="${Record[2]}"
backup_dir="${Record[3]}"
done; then
echo "Could not fech next task metadata from database"
exit 1
fi <<EOF
WITH firsttask AS (SELECT taskid from tasks
WHERE status = 'PENDING'
ORDER BY date_started ASC
LIMIT 1)
SELECT taskid, username, storage_password AS backup_password, location AS backup_dir
FROM firsttask
INNER JOIN users USING (userid)
INNER JOIN storage USING (userid)
WHERE (username = '$1');
EOF
}
For some reason, bash does not detect the last EOF and reports:
./processor.sh: line 138: warning: here-document at line 41 delimited by end-of-file (wanted `EOF')
./processor.sh: line 139: syntax error: unexpected end of file
Any ideas why the EOF is not picked up? Thank you!
It fails because the closing EOF word has trailing whitespace. It must be on a line by itself, with no leading or trailing whitespace.
An exception is when using the <<- syntax, in which case the closing word may be preceded by one or more TABs (but never spaces).
the below script gives me an error. Basically I am trying to delete the records that I got from the first query. I have put them in a text file, formatted them and used them in the delete operation.
After executing the script I am getting the below error:-
: line 5: syntax error at line 27: `<<' unmatched
Can't tell because the code you dumped is unformatted, but my first guess would be you have leading spaces in front of the EOF in your here document.
This should work (note that there are no leading spaces in front of the EOF.:
sqlplus -s $dbcreds << EOF > output.txt
SET SERVEROUTPUT OFF
select empname from emp where dept_no=123;
EOF
if [ -s "output.txt" ]
then
echo " Found the below employees....Deleting them from Database ..............!!!! \n"
cat output.txt
sed "s/(.*)/'\1'/" output.txt| tr '\n' ','|sed 's/.$//' >final_employees.txt
while read line
do
sqlplus -s $dbcreds <<EOF
SET SERVEROUTPUT OFF
Delete from emp where empname in ($line);
EOF
done < final_employees.txt
else
echo " No employees found....!!!"
fi
I have the following code in my script:
RESULT=$(cora_cmd --input={connect hrdwtst01.campbellsci.com';' file-control $STATION_NAME stop-program';' bye';'})
declare y
IFS=$'\n' y=($RESULT)
echo ${y[2]}
if [ ${y[2]} == '-file-control,loggernet datalogger locked' ]; then
echo -e "\t\E[31;1m.. ERROR - Not able to stop the datalogger's program. ..\E[37;0m"
fi
echo "$RESULT"
The compare is not working and it never goes into the if statement. Any ideas?
Results of the set -x:
'++ cora_cmd '--input={connect' 'hrdwtst01.campbellsci.com;' file-control TS_CR850_PB_801 'stop-program;' 'bye;}'
+ RESULT='CoraScript 1, 13, 06 Beta
+connect,"coralib3.dll version 1, 7, 18 Beta"
'file-control,loggernet datalogger locked
+ declare y
+ IFS='
'
+ y=($RESULT)
' echo '-file-control,loggernet datalogger locked
-file-control,loggernet datalogger locked
' '!=' '-file-control,loggernet datalogger locked' ']'
+ echo -e '\t\E[31;1m.. ERROR - Not able to stop the datalogger'\''s program. ..\E[37;0m'
.. ERROR - Not able to stop the datalogger's program. ..'
Always quote your variables unless you know you want word splitting and globbing to be done on the expansion.
if [ "${y[2]}" = '-file-control,loggernet datalogger locked' ]; then
You can also use the built-in [[ syntax, which doesn't do word splitting of variables.
if [[ ${y[2]} = '-file-control,loggernet datalogger locked' ]]; then
So there's a stray ^M on there indicating that your cora_cmd output
includes DOS-style \r\n line endings, and you're only stripping off
the \n, leaving the carriage return character. That's why you're not
getting a match... – twalberg
twalberg correctly diagnosed the problem. The simple solution is of course to include \r in IFS:
IFS=$'\r\n' y=($RESULT)
Including the \r in the test == $'-file-control,loggernet datalogger locked\r' is also possible.