I am trying to insert following data in psql and its showing me weird error.
-bash-4.1$ psql -t -d mydb -c 'insert into ftp_mgr (id, info) values("192.168.1.12", '{"username": "Administrator", "password": "abc456$", "serverAddr":"192.168.1.12"}');'
psql: warning: extra command-line argument "password:" ignored
psql: warning: extra command-line argument "abc456$," ignored
psql: warning: extra command-line argument "serverAddr:192.168.1.12});" ignored
psql: FATAL: role "Administrator," does not exist
This is nothing to do with PostgreSQL. It's a shell quoting problem.
You cannot nest single quotes in bash. So this:
'insert into ftp_mgr (id, info) values("192.168.1.12", '{"username": "Administrator", "password": "abc456$", "serverAddr":"192.168.1.12"}');'
is read up to:
'insert ...'{
and the { is *not part of the quoted string. So the shell tries to intepret {"username": "Administrator", "password": "abc456$", "serverAddr":"192.168.1.12"} as a shell command.
The simplest thing to do here is to use a quoted here-document to avoid the need to deal with ' literal quoting mixing with the shell's quoting:
psql -t -d mydb <<'__END__'
insert into ftp_mgr (id, info) values
("192.168.1.12", '{"username": "Administrator", "password": "abc456$", "serverAddr":"192.168.1.12"}');
__END__
The quotes around the here-document tag are important. If you leave them out then bash still looks for and replaces $variable strings, etc, within the here-document text. That can be really handy, but it's clearly not what you want in this case.
if you must do it with -c you have a couple of options:
Use the shell's string concatenation. Whenever you want a single-quote, end the current single quoted string, write "'", and open a new single quoted string. So you'd write:
'insert ... values(..., '"'"'{"....
Confusing to read, isn't it? The first ' ends the single-quoted shell string. Then "'" is a single quote, quoted by double quotes that're consumed by the shell. Then the next ' begins a new single-quoted string.
That's horrible to read, and it's not much better written as:
'insert ... values(..., '\''{"....
so personally, when I need single quotes within an argument string I either double-quote the whole string and backslash-escape shell metacharacters:
"insert into ftp_mgr (id, info) values(\"192.168.1.12\", '{\"username\": \"Administrator\", \"password\": \"abc456\$\", \"serverAddr\":\"192.168.1.12\"}');"
... or where possible I use a quoted here-document to get around the whole horrifying mess.
My preference is to just avoid the shell for this kind of thing and use a simple Python script.
import psycopg2
conn = psycopg2.connect('dbname=postgres')
curs = conn.cursor()
curs.execute(r'insert into ftp_mgr (id, info) values(%s, %s);', (
r'192.168.1.12',
r'{"username": "Administrator", "password": "abc456$", "serverAddr":"192.168.1.12"}'
))
Python's support for raw-strings (r''), triple quote strings ("""), etc make this sort of thing much easier.
Related
How can I escape double quotes inside a double string in Bash?
For example, in my shell script
#!/bin/bash
dbload="load data local infile \"'gfpoint.csv'\" into table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY \"'\n'\" IGNORE 1 LINES"
I can't get the ENCLOSED BY '\"' with double quote to escape correctly. I can't use single quotes for my variable, because I want to use variable $dbtable.
Use a backslash:
echo "\"" # Prints one " character.
A simple example of escaping quotes in the shell:
$ echo 'abc'\''abc'
abc'abc
$ echo "abc"\""abc"
abc"abc
It's done by finishing an already-opened one ('), placing the escaped one (\'), and then opening another one (').
Alternatively:
$ echo 'abc'"'"'abc'
abc'abc
$ echo "abc"'"'"abc"
abc"abc
It's done by finishing already opened one ('), placing a quote in another quote ("'"), and then opening another one (').
More examples: Escaping single-quotes within single-quoted strings
Keep in mind that you can avoid escaping by using ASCII codes of the characters you need to echo.
Example:
echo -e "This is \x22\x27\x22\x27\x22text\x22\x27\x22\x27\x22"
This is "'"'"text"'"'"
\x22 is the ASCII code (in hex) for double quotes and \x27 for single quotes. Similarly you can echo any character.
I suppose if we try to echo the above string with backslashes, we will need a messy two rows backslashed echo... :)
For variable assignment this is the equivalent:
a=$'This is \x22text\x22'
echo "$a"
# Output:
This is "text"
If the variable is already set by another program, you can still apply double/single quotes with sed or similar tools.
Example:
b="Just another text here"
echo "$b"
Just another text here
sed 's/text/"'\0'"/' <<<"$b" #\0 is a special sed operator
Just another "0" here #this is not what i wanted to be
sed 's/text/\x22\x27\0\x27\x22/' <<<"$b"
Just another "'text'" here #now we are talking. You would normally need a dozen of backslashes to achieve the same result in the normal way.
Bash allows you to place strings adjacently, and they'll just end up being glued together.
So this:
echo "Hello"', world!'
produces
Hello, world!
The trick is to alternate between single and double-quoted strings as required. Unfortunately, it quickly gets very messy. For example:
echo "I like to use" '"double quotes"' "sometimes"
produces
I like to use "double quotes" sometimes
In your example, I would do it something like this:
dbtable=example
dbload='load data local infile "'"'gfpoint.csv'"'" into '"table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '"'"'"' LINES "'TERMINATED BY "'"'\n'"'" IGNORE 1 LINES'
echo $dbload
which produces the following output:
load data local infile "'gfpoint.csv'" into table example FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY "'\n'" IGNORE 1 LINES
It's difficult to see what's going on here, but I can annotate it using Unicode quotes. The following won't work in Bash – it's just for illustration:
dbload=‘load data local infile "’“'gfpoint.csv'”‘" into ’“table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '”‘"’“' LINES ”‘TERMINATED BY "’“'\n'”‘" IGNORE 1 LINES’
The quotes like “ ‘ ’ ” in the above will be interpreted by bash. The quotes like " ' will end up in the resulting variable.
If I give the same treatment to the earlier example, it looks like this:
echo “I like to use” ‘"double quotes"’ “sometimes”
Store the double quote character in a variable:
dqt='"'
echo "Double quotes ${dqt}X${dqt} inside a double quoted string"
Output:
Double quotes "X" inside a double quoted string
Check out printf...
#!/bin/bash
mystr="say \"hi\""
Without using printf
echo -e $mystr
Output: say "hi"
Using printf
echo -e $(printf '%q' $mystr)
Output: say \"hi\"
Make use of $"string".
In this example, it would be,
dbload=$"load data local infile \"'gfpoint.csv'\" into table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY \"'\n'\" IGNORE 1 LINES"
Note (from the man page):
A double-quoted string preceded by a dollar sign ($"string") will cause the string to be translated according to the current locale. If the current locale is C or POSIX, the dollar sign is ignored. If the string is translated and replaced, the replacement is double-quoted.
For use with variables that might contain spaces in you Bash script, use triple quotes inside the main quote, e.g.:
[ "$(date -r """$touchfile""" +%Y%m%d)" -eq "$(date +%Y%m%d)" ]
Add "\" before double quote to escape it, instead of \
#! /bin/csh -f
set dbtable = balabala
set dbload = "load data local infile "\""'gfpoint.csv'"\"" into table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '"\""' LINES TERMINATED BY "\""'\n'"\"" IGNORE 1 LINES"
echo $dbload
# load data local infile "'gfpoint.csv'" into table balabala FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY "''" IGNORE 1 LINES
I have a part of json which looks like below:
{
"openstack": {
"admin": {
"username": "admin",
"password": "password",
"tenant_name": "test"
},
and three environment variables defined like this
auth_url=VALUE1
region_name=VALUE3
endpoint_type=VALUE2
I want to insert 3 lines in the input file just after row 2, so that the output is
{
"openstack": {
"auth_url": VALUE1,
"region_name": VALUE2,
"endpoint_type": VALUE3,
"admin": {
"username": "admin",
"password": "password",
"tenant_name": "test"
},
How it can be done using SED, I tried below
sed -e '3i/\t"auth_url":$auth_url,' -i account_2.json
But it not only adds an extra / at row no 3 but it also doesn't actually replace $auth_url with environment variable as well.
You are are misusing the i command. You have to put backslash after it, not a slash.
Furhtermore, the variable is not expanded since it is in single quotes. Try putting it in double quotes, like this
sed "3i\ \"auth_url\":$AUTH," yourfile
I've read that the insert command wants whatever follows the backslash to be on a newline, which is not the case here, where we have everything on a single line. I guess that's GNU sed's which allows it.
To insert three lines, you can use this
sed "3i\ \"auth_url\":$SHELL\n \"auth_url\":$SHELL\n \"auth_url\":$SHELL" os
And it works well with commas too, since they have no special meaning:
sed "3i\ \"auth_url\":$SHELL,\n \"auth_url\":$SHELL,\n \"auth_url\":$SHELL,"
This question already has answers here:
How to cat <<EOF >> a file containing code?
(5 answers)
Closed 3 years ago.
I am attempting to create a simple function during the startup of a postgres instance. I am doing this by mapping some .sh files to the /docker-entrypoint-initdb.d dir.
I continuously hit a problem with this simple function.
#!/bin/bash
set -e
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
CREATE OR REPLACE FUNCTION hi() RETURNS TEXT AS $$
BEGIN
RETURN 'hi'
END;
$$;
EOSQL
2020-01-29 05:12:30.817 UTC [62] ERROR: syntax error at or near "1" at character 49
2020-01-29 05:12:30.817 UTC [62] STATEMENT: CREATE OR REPLACE FUNCTION hi() RETURNS TEXT AS 1
BEGIN
RETURN 'hi'
END;
ERROR: syntax error at or near "1"
LINE 1: CREATE OR REPLACE FUNCTION hi() RETURNS TEXT AS 1
If I change it to something with actual content:
#!/bin/bash
set -e
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
CREATE OR REPLACE FUNCTION hi() RETURNS TEXT AS $func$
BEGIN
RETURN 'hi'
END;
$func$;
EOSQL
I get another error at the same place:
2020-01-29 05:17:36.161 UTC [62] ERROR: syntax error at or near "$" at character 49
2020-01-29 05:17:36.161 UTC [62] STATEMENT: CREATE OR REPLACE FUNCTION hi() RETURNS TEXT AS $
BEGIN
RETURN 'hi'
END;
ERROR: syntax error at or near "$"
LINE 1: CREATE OR REPLACE FUNCTION hi() RETURNS TEXT AS $
Running with the latest postgres version: 12.1
What precisely is the problem with this function definition and why am I getting this error?
The part of your script from <<-EOSQL to EOSQL is called a "here document", and that functionality is documented at the Bash Reference Manual, §3.6.6 "Here Documents". Per that section:
If any part of word [in your case EOSQL] is quoted, […] the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion, […]
In other words — because you haven't quoted any part of EOSQL, Bash is performing parameter expansion (and other similar substitutions) on the contents of the here-document, which includes replacing $$ with the process-ID, and $func with the empty string, before PostgreSQL sees them.
If you just change <<-EOSQL to <<-'EOSQL', it won't do that.
$func and $$ are variables in bash. The first one is the process ID of the shell instance, which explains the number in your error output; the second is just a normal user-defined variable, presumably empty, so $test$ becomes just $ (as $test gets replaced by an empty string).
You can either escape the dollar sign that bash finds significant, using \$\$ and \$test\$, or you can use <<-'EOSQL' to use single-quote (no variable substitution) semantics on the heredoc.
None of this has to do with PostgreSQL, it's bash doing what it thinks you want.
I need to make a curl POST method with a JSON:
request=`$home/post.curl $url " { \"name\": \"aaaaaaa\", \"message\": \"I NEED TO NOT INTERPRET THIS VAR \${rich_message}\" "`
But the server who reads the request gets "Message=I NEED TO NOT INTERPRET THIS VAR ".
Who can I do please?
If you need the entire JSON string to be passed through literally, put it in single quotes:
request=$($home/post.curl "$url" '{ "name": "aaaaaaa",
"message": "I NEED TO NOT INTERPRET THIS VAR \${rich_message}"}$)
(Notice that I added the missing closing } and changed the obsolescent backticks to the preferred $(...) syntax. I also split the string across multiple lines for legibility. Finally, I put "$url" in double quotes, as this type of string should usually be properly quoted.)
If you require the entire string to be in double quotes, the syntax you were using would require an additional backslash (backticks work similarly to double quotes; you need another escape to escape the escape, as it were); but switching to $(...) fixes that as a side effect, too.
The \ infant of $ is wrong.
Also you are missing } at the end in order to have proper JSON.
So with proper JSON should be:
" { \"name\": \"aaaaaaa\", \"message\": \"I NEED TO NOT INTERPRET THIS VAR ${rich_message}\" }"
PhoneBook.txt has names delimited by :
ex= first:last:number - simple
ex= first first:last last:number - 2 firstnames and/or 2 last names
ex= f'irst:l'ast:number - first name or last name with a single quote in it.
My current script
#!/bin/bash
firstName="$1"
lastName="$2"
if [[ "$firstName" == "" ]]; then
read -p "Please enter a first name: " firstName
fi
if [[ "$lastName" == "" ]]; then
read -p "Please enter a last name: " lastName
fi
grep "$firstName:$lastName" PhoneBook.txt
The -F argument tells grep to search for a fixed string instead of a regex.
Use double quotes to delimit your arguments on the command line, e.g.:
./phonebook "first1 first2" "last o'last"
You're free to use single quotes in a double-quoted string.
Just be aware that double-quoted strings are subject to parameter (variable) and command substitution, so instances of literal $ chars. must be escaped as \$.
Similarly, embedded double quotes must be escaped as \".
Note that, by contrast, there is no direct way to embed single quotes in a single-quoted string - you have to break the string apart and insert an escaped single quote, \'; e.g., to single-quote isn't, use 'isn'\''t'; however, you can freely embed double quotes.