Is there some way to make psql separate the fields and records by \0, aka NUL? It's the only way to be able to pass arbitrary data to Bash scripts.
Based on Matthew Wood's answer, I would expect this to print more that 1 on a newly initialized database:
declare -i count=0
echo "\pset recordsep '\000'
\f '\000'
select typname from pg_type" | \
sudo -iu postgres psql --no-align --quiet --tuples-only -d dbname -U username | while IFS= read -r -d ''
do
#echo "$REPLY"
let count++
done
if [ -n "$REPLY" ]
then
#echo "$REPLY"
let count++
fi
echo $count
Workaround: Iff the SELECT results are unique, you can use this workaround to handle one at a time:
next_record() {
psql --no-align --quiet --tuples-only -d dbname -U username <<SQL
SELECT colname
FROM tablename
WHERE colname > '${1}'
ORDER BY colname
LIMIT 1
SQL
}
last_col=
while true
do
colx="$(next_record "$last_col"; printf x)"
if [ "$colx" = x ]
then
exit
fi
col="${colx%$'\nx'}" # The extra \n character is from psql
# Do your thing here
col_escaped="${col//"'"/''}" # Double single quotes
col_escaped="${col_escaped//\\/\\\\}" # Double backslashes
last_col="$col_escaped"
done
This is not supported. psql uses C print functions to print out the result tables, and printing a zero byte just doesn't work there.
Update: This is now supported in PostgreSQL 9.2-to-be (git).
Try this:
psql --field-separator '\000' --no-align -c '<your query>'
Edit: Maybe not. However, it appear to work in psql using these commands:
\f '\000'
\a
Newer versions of psql support the --field-separator-zero flag.
Related
I am fetching the data from psql in the shell script and assign to the global variable but the global variable is not updating below i have tried:
#!/bin/bash
res_count=0
psql -h $db_host -U $db_user -d $db_name -At -c "select count(id) as dCount from abc" --no-password --field-separator ' ' | \
while read dCount ; do
res_count=$dCount
done;
echo $res_count
$res_count is not updating, it has still value 0, please correct me where i am wrong thanks
Your while loop executes in a subshell because it is executed as part of the pipeline. You can avoid it by using lastpipe or placing the psql command inside process substitution.
#/bin/bash
shopt -s lastpipe
...
Or
res_count=0
while read dCount ; do
res_count=$dCount
done < <(psql -h "$db_host" -U "$db_user" -d "$db_name" -At \
-c "select count(id) as dCount from abc"
--no-password --field-separator ' ')
echo "$res_count"
As a side note, quote your variables properly.
I'm kind of new in bash script and postgresql.
I saw in another question a way to run a bash script as psql user here.
I tried making a bash function as follow,
postgres_create_db(){
sudo su postgres <<- EOF
if psql -lqt | cut -d \| -f 1 | grep -qw nokia_aaa_poc_db; then
psql -c '\dt'
else
psql -c 'CREATE DATABASE nokia_AAA_poc_db;'
fi
EOF
exit
}
where this function will be called further in code, but I wonder if I can add a RETURN to the function that's actualy returning a varible that was first declared inside postgres bash (in between the EOF's). Like bellow:
postgres_create_db(){
sudo su postgres <<- EOF
if psql -lqt | cut -d \| -f 1 | grep -qw nokia_aaa_poc_db; then
psql -c '\dt'
exists=1 #where thats a variable that I want to access outside the postgres bash.
else
psql -c 'CREATE DATABASE nokia_AAA_poc_db;'
fi
EOF
exit
return exists
}
but it gives an error on shellcheck
return exists
^-- SC2152: Can only return 0-255. Other data should be written to stdout.
Functions in bash can only return values from 0 to 255 where 0 is success. Reference: Return value in a Bash function
So you can echo the variable like this instead:
#!/usr/bin/env bash
postgres_test() {
psql -c '\dt' &> /dev/null
declare exists=1
echo $exists
}
printf "%s\n" "$(postgres_test)"
This prints "1".
You'll also notice that I redirected the output of the Postgres command to /dev/null. This is because it would be combined in the function's output otherwise.
You might wish to redirect that output to a file instead.
Trying it this way:
#!/bin/bash
myvals=`psql -d mydb -c "select id from table1 where 't'"`
ssh user1#host1.domain.tld "for i in $myvals; do echo \$i >> values; done"
As long as psql returns just one value, it works fine. But if its several values, I receive this response:
bash: -c: line 1: syntax error near unexpected token `2'
bash: -c: line 1: `2'
Also, I tried to:
myvals='1 2 3'
And then it works fine: the values 1 2 3 are appended to the "values" file on the remote host; no error mesages.
If I try another subshell command, such as myvals=ls /bin, errors reappear.
It's clear that $myvals is evaluated on the local host already but what makes the subshell results so different?
If It's Not Really An Array...
Iterating over a string as if it were an array is innately buggy. Don't do it. That said, to generate a safely-escaped (eval-safe) version of your value, use printf %q.
#!/bin/bash
myvals=`psql -d mydb -c "select id from table1 where 't'"`
printf -v myvals_q %q "$myvals"
ssh user1#host1.domain.tld \
"myvals=$myvals_q;"' for i in $myvals; do echo "$i"; done >>values'
If You Actually Had An Array
#!/bin/bash
readarray -t myvals < <(psql -d mydb -c "select id from table1 where 't'")
printf -v myvals_q '%q ' "${myvals[#]}"
ssh user1#host1.domain.tld \
"myvals=( $myvals_q );"' for i in "${myvals[#]}"; do echo "$i"; done >>values'
If You Don't Need To Store The Value Locally In The First Place
#!/bin/bash
ssh user1#host1.domain.tld \
'while read -r i; do echo "$i"; done >>values' \
< <(psql -d mydb -c "select id from table1 where 't'")
General Notes
Running echo "$i" >>values over and over in a loop is inefficient: Every time the line is run, it re-opens the values file. Instead, run the redirection >values over the whole loop; this truncates the file exactly once, at the loop's start, and appends all values generated therein.
Unquoted expansions are generally dangerous. For example, if foo='*', then $foo will be replaced with a list of files in the current directory, but "$foo" will emit the exact contents -- *. Similarly, tabs, whitespace runs, and various other contents can be unintentionally damaged by unquoted expansion, even when passing directly to echo.
You can switch quoting types in the same string -- thus, "$foo"'$foo' is one string, the first part of which is replaced with the value of the variable named foo, and the second component of which is the exact string $foo.
You can send the output as a file:
#!/bin/bash
psql -d mydb -c "select id from table1 where 't'" > /tmp/values
scp values user1#host1.domain.tld:/tmp/
or pipe it to the remote host:
psql -d mydb -c "select id from table1 where 't'" | \
ssh user1#host1.domain.tld 'while read line; do echo $line; done'
I'm having difficulty getting the correct variable substitution to work within this function, especially the use of echo "$(...) string.
outputOFF ()
{
host='mydevreporting.com'
_pw='123456foobar'
_dt=$(date +'%m-%d-%y')
exp="SELECT * FROM metrics.account_use where account='foo' and profile='bar' order by date desc;";
echo "$(grep real < <({ time mysql -u admin -p${_pw} -h ${host} -N -e "$exp"} 2>&1)):localhost:${_dt}"
#echo "$(grep real < <(\{ time mysql -u admin -p${_pw} -h ${host} -N -e "$exp"\} 2>&1)):localhost:${_dt}"
}
From the command line, it will work:
echo "$(grep real < <({ time mysql -u admin -p123456foobar -h mydevreporting.com -N -e "SELECT * FROM metrics.account_use where account='foo' and profile='bar' order by date desc;"; } 2>&1)):localhost:$(date +'%m-%d-%y')"
As I'm seeing error messages:
./tst.sh: command substitution: line 40: syntax error near unexpected token `)'
./tst.sh: command substitution: line 40: `{ time mysql -u admin -p${_pw} -h ${host} -N -e "$exp"} 2>&1)'
As you can see the echo "$(...) is inside this function outputOFF().
I've also tried escaping the braces \{, \}, which allows the variables to substitute, but somehow that command isn't working as it should.
echo "$(grep real < <(\{ time mysql -u admin -p${_pw} -h ${host} -N -e "$exp"\} 2>&1)):localhost:${_dt}"
So, i'm stuck.
You are missing a ; in the {...} group expression, after "$exp" (see this documentation for why). Here is the corrected version:
echo "$(grep real < <({ time mysql -u admin -p${_pw} -h ${host} -N -e "$exp"; } 2>&1)):localhost:${_dt}"
Less braces makes it more readable, at least for me.
result=$( { time mysql -u admin -p${_pw} -h ${host} -N -e "$exp"; } 2>&1 | grep real );
echo "${result}:localhost:${_dt}"
time is reporting on stderr. Thus the {} are neceessary to capture the output.
Or discarding the result, and only capture the result of time.
result=$( { time mysql -u admin -p${_pw} -h ${host} -N -e "$exp" >/dev/null; } 2>&1 )
echo ${result}":localhost:${_dt}"
The unquoted ${result} is printed without the newlines. Thus you can keep all information from time with the additional timestamp.
is there any way, by which I can generate a DDL of GP Functions
My actual requirement is I want to create/generate DDL of each objects available in particular schema in separate SQL file
I am able to do it for tables and views via below script
tbl_list=`psql -At -c "select tablename from pg_tables where schemaname ='${PGSCHEMA}' and tablename not like '%_prt_%' order by 1;"` # Fetch all table Name
for fname in $tbl_list
do
ddl=`PGPASSWORD='passwd' pg_dump -h 10.128.19.297 -U gpadmin jiodata -s -t "${PGSCHEMA}.$fname" >${script_dir}/${output_dir}/$fname.sql` # Fetch ddl for all tables
#pg_dump -h 10.128.19.297 -U gpadmin jiodata -s -t "${PGSCHEMA}.$fname" >${script_dir}/${output_dir}/$fname.sql
echo "Table DDL generated : "${PGSCHEMA}.$fname | tee -a ${log_file}
done
Did any one tried it for Functions
Please help if anyone know
Thanks
Create a sql file named "get_functions.sql" with this code:
select sub.function_name || '(' || array_to_string(array_agg(typname), ',') || ')'
from (
select n.nspname as schema_name, proname as function_name, unnest(proallargtypes) as parm_type, unnest(proargmodes) as parm_direction
from pg_proc p
join pg_namespace n on p.pronamespace = n.oid
where n.nspname = :schema_name
) as sub
join pg_type t on sub.parm_type = t.oid
where sub.parm_direction = 'i'
group by sub.function_name;
Next, create a bash script with this:
#!/bin/bash
set -e
schema_name="$1"
if [ "$schema_name" == "" ]; then
echo "ERROR: You must provide the schema name."
echo "Example usage: ./runme.sh gp_toolkit"
exit 1
fi
echo "pg_dump -Fc -s -n $schema_name > $schema_name.sql"
pg_dump -Fc -s -n $schema_name > $schema_name.sql
for i in $(psql -t -A -v schema_name="'$schema_name'" -f get_functions.sql); do
filename=$(echo $i | tr \( _ | tr \) _)
filename+=".sql"
echo "cat $schema_name.sql | pg_restore -P '$i' > $filename"
cat $schema_name.sql | pg_restore -P ''$i'' > $filename
done
Fix the permissions on the script.
chmod 755 runme.sh
And execute the script.
./runme.sh gp_toolkit
This will create a file per function and it allows for overloaded functions too.