My question is a simple one. I am looking to simply format my bash code into something a little more readable for other users .. While this works:
mysql --login-path=main-data -N -e "SELECT section_id FROM db.table WHERE contractor_id = '1'" | while read contractor_info_id; do
#......//
done
I don't understand why the backslash doesn't work with a input output statement separated by | .. IE
mysql --login-path=main-data -N -e "SELECT section_id\
FROM db.table\
WHERE contractor_id = '1'" | while read contractor_info_id; do
#......//
done
It generates a fatal error Syntax error: "done" unexpected
Why does multi-line formatting work on a single output command such as:
long_arg="my very long string\
which does not fit\
on the screen"
But not on an output | input command ?
The reason the escaped line ending does not work in your mysql use case is that mysql does not require an escaped line ending. The text between the two double quotes is treated as the string, and escaped characters are no interpreted. (eg: \t would not put in a tab). You can se this in action with these examples:
$ echo "Hello from line one
> and hello from line two"
Hello from line one
and hello from line two
$ echo Hello from line one \
> and also from line one
Hello from line one and also from line one
TL;DR: Within double-quotes, the slash and CR is treated as part of the string, not interpreted as an escape character.
Backslashes not needed inside quotes, and SQL isn't usually picky about extra whitespace anyway.
I don't have mysql installed here, but try this and let me know if it doesn't behave...
mysql --login-path=main-data -N -e "
SELECT section_id
FROM db.table
WHERE contractor_id = '1'
" | while read contractor_info_id
do echo $contractor_info_id
done
mysql "..." | while read contractor_info_id; do
#......//
done
Syntax error: "done" unexpected
This error is coming from bash, not from mysql, so we can rule out anything inside the double quotes. It doesn't matter what you're doing with backslashes there.
Do you actually have anything in place of #......//? You need at least one command inside the loop, otherwise you'll get this "done" unexpected error. Try :, a no-op command. That'll at least get rid of the syntax error.
mysql "..." | while read contractor_info_id; do
:
done
"SELECT section_id\
FROM db.table\
WHERE contractor_id = '1'"
The backslashes won't cause bash any heartburn but they do lead to invalid SQL. Once the shell deletes the backslashes and newlines you're left with:
"SELECT section_idFROM db.tableWHERE contractor_id = '1'"
To fix that you can either add spaces, or just leave out the backslashes. MySQL doesn't care if there are newlines, nor does bash, so you can stick with what you wrote originally.
"SELECT section_id
FROM db.table
WHERE contractor_id = '1'"
The reason for this happening to me was a unique issue I was having with my IDE .. It was inserting an actual character for a line break rather than just a physical line-break itself. The problem wasn't the syntax I was attempting, rather the inserted characters on save. Thanks to all who thoughtfully answered my question. My original un-escaped syntax was correct to begin with.
Related
I need my script to send an email from terminal. Based on what I've seen here and many other places online, I formatted it like this:
/var/mail -s "$SUBJECT" "$EMAIL" << EOF
Here's a line of my message!
And here's another line!
Last line of the message here!
EOF
However, when I run this I get this warning:
myfile.sh: line x: warning: here-document at line y delimited by end-of-file (wanted 'EOF')
myfile.sh: line x+1: syntax error: unexpected end of file
...where line x is the last written line of code in the program, and line y is the line with /var/mail in it. I've tried replacing EOF with other things (ENDOFMESSAGE, FINISH, etc.) but to no avail. Nearly everything I've found online has it done this way, and I'm really new at bash so I'm having a hard time figuring it out on my own. Could anyone offer any help?
The EOF token must be at the beginning of the line, you can't indent it along with the block of code it goes with.
If you write <<-EOF you may indent it, but it must be indented with Tab characters, not spaces. So it still might not end up even with the block of code.
Also make sure you have no whitespace after the EOF token on the line.
The line that starts or ends the here-doc probably has some non-printable or whitespace characters (for example, carriage return) which means that the second "EOF" does not match the first, and doesn't end the here-doc like it should. This is a very common error, and difficult to detect with just a text editor. You can make non-printable characters visible for example with cat:
cat -A myfile.sh
Once you see the output from cat -A the solution will be obvious: remove the offending characters.
Please try to remove the preceeding spaces before EOF:-
/var/mail -s "$SUBJECT" "$EMAIL" <<-EOF
Using <tab> instead of <spaces> for ident AND using <<-EOF works fine.
The "-" removes the <tabs>, not <spaces>, but at least this works.
Note one can also get this error if you do this;
while read line; do
echo $line
done << somefile
Because << somefile should read < somefile in this case.
May be old but I had a space after the ending EOF
<< EOF
blah
blah
EOF <-- this was the issue. Had it for years, finally looked it up here
For anyone stumbling here who googled "bash warning: here-document delimited by end-of-file", it may be that you are getting the
warning: here-document at line 74 delimited by end-of-file
...type warning because you accidentally used a here document symbol (<<) when you meant to use a here string symbol (<<<). That was my case.
Here is a flexible way to do deal with multiple indented lines without using heredoc.
echo 'Hello!'
sed -e 's:^\s*::' < <(echo '
Some indented text here.
Some indented text here.
')
if [[ true ]]; then
sed -e 's:^\s\{4,4\}::' < <(echo '
Some indented text here.
Some extra indented text here.
Some indented text here.
')
fi
Some notes on this solution:
if the content is expected to have simple quotes, either escape them using \ or replace the string delimiters with double quotes. In the latter case, be careful that construction like $(command) will be interpreted. If the string contains both simple and double quotes, you'll have to escape at least of kind.
the given example print a trailing empty line, there are numerous way to get rid of it, not included here to keep the proposal to a minimum clutter
the flexibility comes from the ease with which you can control how much leading space should stay or go, provided that you know some sed REGEXP of course.
When I want to have docstrings for my bash functions, I use a solution similar to the suggestion of user12205 in a duplicate of this question.
See how I define USAGE for a solution that:
auto-formats well for me in my IDE of choice (sublime)
is multi-line
can use spaces or tabs as indentation
preserves indentations within the comment.
function foo {
# Docstring
read -r -d '' USAGE <<' END'
# This method prints foo to the terminal.
#
# Enter `foo -h` to see the docstring.
# It has indentations and multiple lines.
#
# Change the delimiter if you need hashtag for some reason.
# This can include $$ and = and eval, but won't be evaluated
END
if [ "$1" = "-h" ]
then
echo "$USAGE" | cut -d "#" -f 2 | cut -c 2-
return
fi
echo "foo"
}
So foo -h yields:
This method prints foo to the terminal.
Enter `foo -h` to see the docstring.
It has indentations and multiple lines.
Change the delimiter if you need hashtag for some reason.
This can include $$ and = and eval, but won't be evaluated
Explanation
cut -d "#" -f 2: Retrieve the second portion of the # delimited lines. (Think a csv with "#" as the delimiter, empty first column).
cut -c 2-: Retrieve the 2nd to end character of the resultant string
Also note that if [ "$1" = "-h" ] evaluates as False if there is no first argument, w/o error, since it becomes an empty string.
make sure where you put the ending EOF you put it at the beginning of a new line
Along with the other answers mentioned by Barmar and Joni, I've noticed that I sometimes have to leave a blank line before and after my EOF when using <<-EOF.
I need my script to send an email from terminal. Based on what I've seen here and many other places online, I formatted it like this:
/var/mail -s "$SUBJECT" "$EMAIL" << EOF
Here's a line of my message!
And here's another line!
Last line of the message here!
EOF
However, when I run this I get this warning:
myfile.sh: line x: warning: here-document at line y delimited by end-of-file (wanted 'EOF')
myfile.sh: line x+1: syntax error: unexpected end of file
...where line x is the last written line of code in the program, and line y is the line with /var/mail in it. I've tried replacing EOF with other things (ENDOFMESSAGE, FINISH, etc.) but to no avail. Nearly everything I've found online has it done this way, and I'm really new at bash so I'm having a hard time figuring it out on my own. Could anyone offer any help?
The EOF token must be at the beginning of the line, you can't indent it along with the block of code it goes with.
If you write <<-EOF you may indent it, but it must be indented with Tab characters, not spaces. So it still might not end up even with the block of code.
Also make sure you have no whitespace after the EOF token on the line.
The line that starts or ends the here-doc probably has some non-printable or whitespace characters (for example, carriage return) which means that the second "EOF" does not match the first, and doesn't end the here-doc like it should. This is a very common error, and difficult to detect with just a text editor. You can make non-printable characters visible for example with cat:
cat -A myfile.sh
Once you see the output from cat -A the solution will be obvious: remove the offending characters.
Please try to remove the preceeding spaces before EOF:-
/var/mail -s "$SUBJECT" "$EMAIL" <<-EOF
Using <tab> instead of <spaces> for ident AND using <<-EOF works fine.
The "-" removes the <tabs>, not <spaces>, but at least this works.
Note one can also get this error if you do this;
while read line; do
echo $line
done << somefile
Because << somefile should read < somefile in this case.
May be old but I had a space after the ending EOF
<< EOF
blah
blah
EOF <-- this was the issue. Had it for years, finally looked it up here
For anyone stumbling here who googled "bash warning: here-document delimited by end-of-file", it may be that you are getting the
warning: here-document at line 74 delimited by end-of-file
...type warning because you accidentally used a here document symbol (<<) when you meant to use a here string symbol (<<<). That was my case.
Here is a flexible way to do deal with multiple indented lines without using heredoc.
echo 'Hello!'
sed -e 's:^\s*::' < <(echo '
Some indented text here.
Some indented text here.
')
if [[ true ]]; then
sed -e 's:^\s\{4,4\}::' < <(echo '
Some indented text here.
Some extra indented text here.
Some indented text here.
')
fi
Some notes on this solution:
if the content is expected to have simple quotes, either escape them using \ or replace the string delimiters with double quotes. In the latter case, be careful that construction like $(command) will be interpreted. If the string contains both simple and double quotes, you'll have to escape at least of kind.
the given example print a trailing empty line, there are numerous way to get rid of it, not included here to keep the proposal to a minimum clutter
the flexibility comes from the ease with which you can control how much leading space should stay or go, provided that you know some sed REGEXP of course.
When I want to have docstrings for my bash functions, I use a solution similar to the suggestion of user12205 in a duplicate of this question.
See how I define USAGE for a solution that:
auto-formats well for me in my IDE of choice (sublime)
is multi-line
can use spaces or tabs as indentation
preserves indentations within the comment.
function foo {
# Docstring
read -r -d '' USAGE <<' END'
# This method prints foo to the terminal.
#
# Enter `foo -h` to see the docstring.
# It has indentations and multiple lines.
#
# Change the delimiter if you need hashtag for some reason.
# This can include $$ and = and eval, but won't be evaluated
END
if [ "$1" = "-h" ]
then
echo "$USAGE" | cut -d "#" -f 2 | cut -c 2-
return
fi
echo "foo"
}
So foo -h yields:
This method prints foo to the terminal.
Enter `foo -h` to see the docstring.
It has indentations and multiple lines.
Change the delimiter if you need hashtag for some reason.
This can include $$ and = and eval, but won't be evaluated
Explanation
cut -d "#" -f 2: Retrieve the second portion of the # delimited lines. (Think a csv with "#" as the delimiter, empty first column).
cut -c 2-: Retrieve the 2nd to end character of the resultant string
Also note that if [ "$1" = "-h" ] evaluates as False if there is no first argument, w/o error, since it becomes an empty string.
make sure where you put the ending EOF you put it at the beginning of a new line
Along with the other answers mentioned by Barmar and Joni, I've noticed that I sometimes have to leave a blank line before and after my EOF when using <<-EOF.
I try to substitute a line in my .sql file with the sed command.
I encountered an issue when i try to "escape" the last parenthesis of the line.
I tried to escape it with a backlash , used simple and double quotes around it but the problem persist.
sed -i "s|\(select dbms_metadata.get_ddl('PACKAGE',\).*|\1"'${var}','PRC_BNE_JZ') from dual;"|" backup_script.sql
The actual result is the following error :
syntax error near unexpected token `)'
The expected result is to understand my error.
Untested since you didn't provide any sample input/output to test against but this might be what you're looking for:
sed -i 's/\(select dbms_metadata.get_ddl('\''PACKAGE'\'',\).*/\1'"${var}"','\''PRC_BNE_JZ'\'') from dual;/' backup_script.sql
Don't use | as a delimiter since it's an ERE metacharacter and so makes your code confusing to read at best and can cause problems if/when you decide to use EREs later (via the -E arg).
Do always use single quotes around scripts and strings unless you have a very specific reason why you must not do so and fully understand all the implications.
I do everything in single quotes, to prevent strange expansions. To have a single quote in the string, I use '\'' as in ' string '\'' rest of string '. To have a variable I use '"$var"' as in ' string '"$var"' rest of string to have it properly expanded and concatenated with the rest of the string.
The following works:
> var=var
> echo "select dbms_metadata.get_ddl('PACKAGE'," |
> sed 's|\(select dbms_metadata.get_ddl('\''PACKAGE'\'',\).*|\1'\'"${var}"\'','\''PRC_BNE_JZ'\'') from dual;|'
select dbms_metadata.get_ddl('PACKAGE','var','PRC_BNE_JZ') from dual;
But probably using " is probably easier in this case, as the string uses ' everywhere, as in:
sed "s|\(select dbms_metadata.get_ddl('PACKAGE',\).*|\1'${var}','PRC_BNE_JZ') from dual;|"
The error comes from the shell that can't parse the arguments. Ex. for the following:
> echo abc)
main.sh: line 2: syntax error near unexpected token `)'
main.sh: line 2: `echo abc)'
The following happens in your command:
sed -i " bla bla "'${var}','PRC_BNE_JZ') from dual;"|" backup_script.sql
^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^ - could be 4th argument
^ - unquoted `)` is parsed by bash, as in subshell `( ... )`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - 2nd argument
^^ - 1st argument to sed command
^^^ - run sed command
I need my script to send an email from terminal. Based on what I've seen here and many other places online, I formatted it like this:
/var/mail -s "$SUBJECT" "$EMAIL" << EOF
Here's a line of my message!
And here's another line!
Last line of the message here!
EOF
However, when I run this I get this warning:
myfile.sh: line x: warning: here-document at line y delimited by end-of-file (wanted 'EOF')
myfile.sh: line x+1: syntax error: unexpected end of file
...where line x is the last written line of code in the program, and line y is the line with /var/mail in it. I've tried replacing EOF with other things (ENDOFMESSAGE, FINISH, etc.) but to no avail. Nearly everything I've found online has it done this way, and I'm really new at bash so I'm having a hard time figuring it out on my own. Could anyone offer any help?
The EOF token must be at the beginning of the line, you can't indent it along with the block of code it goes with.
If you write <<-EOF you may indent it, but it must be indented with Tab characters, not spaces. So it still might not end up even with the block of code.
Also make sure you have no whitespace after the EOF token on the line.
The line that starts or ends the here-doc probably has some non-printable or whitespace characters (for example, carriage return) which means that the second "EOF" does not match the first, and doesn't end the here-doc like it should. This is a very common error, and difficult to detect with just a text editor. You can make non-printable characters visible for example with cat:
cat -A myfile.sh
Once you see the output from cat -A the solution will be obvious: remove the offending characters.
Please try to remove the preceeding spaces before EOF:-
/var/mail -s "$SUBJECT" "$EMAIL" <<-EOF
Using <tab> instead of <spaces> for ident AND using <<-EOF works fine.
The "-" removes the <tabs>, not <spaces>, but at least this works.
Note one can also get this error if you do this;
while read line; do
echo $line
done << somefile
Because << somefile should read < somefile in this case.
May be old but I had a space after the ending EOF
<< EOF
blah
blah
EOF <-- this was the issue. Had it for years, finally looked it up here
For anyone stumbling here who googled "bash warning: here-document delimited by end-of-file", it may be that you are getting the
warning: here-document at line 74 delimited by end-of-file
...type warning because you accidentally used a here document symbol (<<) when you meant to use a here string symbol (<<<). That was my case.
Here is a flexible way to do deal with multiple indented lines without using heredoc.
echo 'Hello!'
sed -e 's:^\s*::' < <(echo '
Some indented text here.
Some indented text here.
')
if [[ true ]]; then
sed -e 's:^\s\{4,4\}::' < <(echo '
Some indented text here.
Some extra indented text here.
Some indented text here.
')
fi
Some notes on this solution:
if the content is expected to have simple quotes, either escape them using \ or replace the string delimiters with double quotes. In the latter case, be careful that construction like $(command) will be interpreted. If the string contains both simple and double quotes, you'll have to escape at least of kind.
the given example print a trailing empty line, there are numerous way to get rid of it, not included here to keep the proposal to a minimum clutter
the flexibility comes from the ease with which you can control how much leading space should stay or go, provided that you know some sed REGEXP of course.
When I want to have docstrings for my bash functions, I use a solution similar to the suggestion of user12205 in a duplicate of this question.
See how I define USAGE for a solution that:
auto-formats well for me in my IDE of choice (sublime)
is multi-line
can use spaces or tabs as indentation
preserves indentations within the comment.
function foo {
# Docstring
read -r -d '' USAGE <<' END'
# This method prints foo to the terminal.
#
# Enter `foo -h` to see the docstring.
# It has indentations and multiple lines.
#
# Change the delimiter if you need hashtag for some reason.
# This can include $$ and = and eval, but won't be evaluated
END
if [ "$1" = "-h" ]
then
echo "$USAGE" | cut -d "#" -f 2 | cut -c 2-
return
fi
echo "foo"
}
So foo -h yields:
This method prints foo to the terminal.
Enter `foo -h` to see the docstring.
It has indentations and multiple lines.
Change the delimiter if you need hashtag for some reason.
This can include $$ and = and eval, but won't be evaluated
Explanation
cut -d "#" -f 2: Retrieve the second portion of the # delimited lines. (Think a csv with "#" as the delimiter, empty first column).
cut -c 2-: Retrieve the 2nd to end character of the resultant string
Also note that if [ "$1" = "-h" ] evaluates as False if there is no first argument, w/o error, since it becomes an empty string.
make sure where you put the ending EOF you put it at the beginning of a new line
Along with the other answers mentioned by Barmar and Joni, I've noticed that I sometimes have to leave a blank line before and after my EOF when using <<-EOF.
My original issue was to be able to add a line at the end of a specific block in a configuration file.
############
# MY BLOCK #
############
VALUE1 = XXXXX
VALUE2 = YYYYY
MYNEWVALUE = XXXXX <<< I want to add this one
##############
# MY BLOCK 2 #
##############
To do this I used the following sed script and it work flawlessly (found it in another post) :
sed -i -e "/# MY BLOCK #/{:a;n;/^$/!ba;i\MYNEWVALUE = XXXXX" -e '}' myfile
This worked perfectly when executed inside a shell script but I can't manage to use it directly in an interactive shell (it gave me an error: "!ba event not found"). To solve this, I tried to add '\' before '!ba' but now it gave me another error which tells me that '\' is an unknown command.
Could anyone explain where my mistake is on the above issue and how this script works?
Here is my understanding:
-i : insert new line (i think the first one is useless, am i right?)
-e : execute this sed script (don't understand why there is a second one at the end to close the })
:a : begin a loop
n : read each line with the pattern ^$ (empty lines)
! : reverse the loop
ba : end of the loop
Thanks !
Use ' instead of " to avoid having bash try to do history substitution on the !
If XXXXX contains a shell parameter expansion or somesuch, you can do it like this:
sed -i -e"/# $BLOCK_NAME"'#/{:a;n;/^$/!ba;i\'"$NEW_VAR = $NEW_VALUE" -e"}" myfile
The second -e is required to effectively insert a newline to close off the i command. You could actually insert the newline directly, instead:
sed -i -e"/# $BLOCK_NAME "'#/{:a;n;/^$/!ba;i\'"$NEW_VAR = $NEW_VALUE"$'\n}' myfile
:a introduces a label, named a.
n writes current pattern space to output, and replace pattern space with next line of input.
/^$/! means to match lines that are NOT (!) blank lines in pattern space; the following ba is a "branch to label a" when that match (not blank line) occurs.
If the branch doesn't occur, the i insert then takes place.
Use single quotes (') instead of double quotes (") on command line to prevent shell from performing shell substitutions (including the "$" and "!" characters).
In interactive shells, ! is used for history substitution, so you need to escape it:
sed -i -e "/# MY BLOCK #/{:a;n;/^\$/\!ba;i\MYNEWVALUE = XXXXX" -e '}' myfile
You should also escape $, since it has special meaning inside doublequoted strings (although in this case it's OK, because it's followed by /, not a variable name).