in Greenplum generate DDL of database objects - greenplum

is there any way, by which I can generate a DDL of GP Functions
My actual requirement is I want to create/generate DDL of each objects available in particular schema in separate SQL file
I am able to do it for tables and views via below script
tbl_list=`psql -At -c "select tablename from pg_tables where schemaname ='${PGSCHEMA}' and tablename not like '%_prt_%' order by 1;"` # Fetch all table Name
for fname in $tbl_list
do
ddl=`PGPASSWORD='passwd' pg_dump -h 10.128.19.297 -U gpadmin jiodata -s -t "${PGSCHEMA}.$fname" >${script_dir}/${output_dir}/$fname.sql` # Fetch ddl for all tables
#pg_dump -h 10.128.19.297 -U gpadmin jiodata -s -t "${PGSCHEMA}.$fname" >${script_dir}/${output_dir}/$fname.sql
echo "Table DDL generated : "${PGSCHEMA}.$fname | tee -a ${log_file}
done
Did any one tried it for Functions
Please help if anyone know
Thanks

Create a sql file named "get_functions.sql" with this code:
select sub.function_name || '(' || array_to_string(array_agg(typname), ',') || ')'
from (
select n.nspname as schema_name, proname as function_name, unnest(proallargtypes) as parm_type, unnest(proargmodes) as parm_direction
from pg_proc p
join pg_namespace n on p.pronamespace = n.oid
where n.nspname = :schema_name
) as sub
join pg_type t on sub.parm_type = t.oid
where sub.parm_direction = 'i'
group by sub.function_name;
Next, create a bash script with this:
#!/bin/bash
set -e
schema_name="$1"
if [ "$schema_name" == "" ]; then
echo "ERROR: You must provide the schema name."
echo "Example usage: ./runme.sh gp_toolkit"
exit 1
fi
echo "pg_dump -Fc -s -n $schema_name > $schema_name.sql"
pg_dump -Fc -s -n $schema_name > $schema_name.sql
for i in $(psql -t -A -v schema_name="'$schema_name'" -f get_functions.sql); do
filename=$(echo $i | tr \( _ | tr \) _)
filename+=".sql"
echo "cat $schema_name.sql | pg_restore -P '$i' > $filename"
cat $schema_name.sql | pg_restore -P ''$i'' > $filename
done
Fix the permissions on the script.
chmod 755 runme.sh
And execute the script.
./runme.sh gp_toolkit
This will create a file per function and it allows for overloaded functions too.

Related

while loop with global variable scope issue in shell script with psql

I am fetching the data from psql in the shell script and assign to the global variable but the global variable is not updating below i have tried:
#!/bin/bash
res_count=0
psql -h $db_host -U $db_user -d $db_name -At -c "select count(id) as dCount from abc" --no-password --field-separator ' ' | \
while read dCount ; do
res_count=$dCount
done;
echo $res_count
$res_count is not updating, it has still value 0, please correct me where i am wrong thanks
Your while loop executes in a subshell because it is executed as part of the pipeline. You can avoid it by using lastpipe or placing the psql command inside process substitution.
#/bin/bash
shopt -s lastpipe
...
Or
res_count=0
while read dCount ; do
res_count=$dCount
done < <(psql -h "$db_host" -U "$db_user" -d "$db_name" -At \
-c "select count(id) as dCount from abc"
--no-password --field-separator ' ')
echo "$res_count"
As a side note, quote your variables properly.

How to delete multi level partition in Hadoop HDFS

Have a multi level partitioned Hive table,now need to delete the partitioned folders which are older
than certain years.
Multilevel partitions looks as below.
/data/warehouse/suite/catalyst/site/company=abc/year=2019/month=08
/data/warehouse/suite/catalyst/site/company=cde/year=2018/month=05
/data/warehouse/suite/catalyst/site/company=cde/year=2017/month=11
/data/warehouse/suite/catalyst/site/company=cde/year=2016/month=11
If want to delete the partitions older than 2 years, That means /year=2017/month=11 and year=2016/month=11 need to be deleted How it can be done.
Pls help, Thanks in advance.
ALTER TABLE mytable drop if exists partition (year<='2017')
You can not control the partition deletion as you are expecting
You can try it using the unix way that is more reliable.
hive -S -e "show partitions test" > tmp.txt
curr_year=`expr "$(date +'%Y')" - "2"`
curr_mon=`expr "$(date +'%m')" - "1"`
cur_part=$curr_year$curr_mon
cur_part=201812
echo $cur_part
#echo "year=2016/month=12" | cut -d '=' -f 2 | grep -o -E '[0-9]+'
#echo "year=2016/month=12" | cut -d '=' -f 4 | grep -o -E '[0-9]+'
while read -r line
do
part_year=`echo $line | cut -d '=' -f 2 | grep -o -E '[0-9]+'`
part_mon=`echo $line | cut -d '=' -f 3 | grep -o -E '[0-9]+'`
part_part=$part_year$part_mon
echo $part_part
if [[ $part_part -lt $cur_part ]]
then
echo "$part_year , $part_mon"
hive --hivevar year="$part_year" --hivevar month="$part_mon" -e 'ALTER TABLE test DROP IF EXISTS PARTITION (year="${hivevar:year}", month="${hivevar:month}")'
fi
done < tmp.txt
> show partitions test;
OK
year=2016/month=12
year=2017/month=11
year=2017/month=12
year=2018/month=12
> show partitions test;
OK
year=2017/month=12
year=2018/month=12
i have tested it is working fine

for loop with 2 files consecutively and pass parameters to other script

I have 2 files, one with hostnames.txt and one with commands.txt :
hostnames.txt:
switch1.txt
switch2.txt
switch3.txt
commands.txt:
show inter gi0/1
show inter gi0/0/1
show inter Eth1/1
I would like to execute switch1 ( first switch ) with show inter gi0/1 ( first command ) and then pick up switch2 and execute with show inter gi0/0/1 and so on until the file ends.
I'm using a TCL script to which I'm passing the parameters with hostname and command from a text files.
for in `/bin/cat hostname.list`;
do
echo $n > oneswitch.txt
for in `/bin/cat interfinal.list`;
do
echo $m > onecommand.txt
for switch in `/bin/cat oneswitch.txt
tclscript -u username -p password -t $switch -r onecommand.txt .
And I couldn't achieve it, how and what are the possible loops I can use and what login can I put in place to get this achieved?
What happens with the above one is that I can only execute with last switch with last command.
Please help.
Thanks
Use paste -> try this at a bash prompt: paste -d ":" hostname.list interfinal.list
So:
paste -d ":" hostname.list interfinal.list | while IFS=: read -r switch cmd; do
tclscript -u username -p password -t "$switch" -r "$cmd" .
done
You could do:
#!/usr/bin/env bash
for i in {1..3}; do
eval $(paste -d' ' <(sed -n ${i}p commands.txt) <(sed -n ${i}p hostnames.txt))
done
This script will execute the commands:
show inter gi0/1 switch1.txt
show inter gi0/0/1 switch2.txt
show inter Eth1/1 switch3.txt

Bash script help/evaluation

I'm trying to learn some scripting however I can't find solution for one functionality.
Basically I would like to ask to evaluate my script as it's probably possible to reduce the complexity and number of lines.
The purpose of this script is to download random, encrypted MySQL backups from Amazon S3, restore the dump and run some random MySQL queries.
I'm not sure how to email the output from printf statements - one is for headers and second one for actual data. I've tried to format the output so it looks like below but I had to exclude the headers from the loop:
Database: Table: Entries:
database1 random_table 0
database2 random_table 0
database3 random_table 0
database4 random_table 0
I would like to include this output in the email and also change the email subject based on the success/failure of the script.
I probably use to much if loops and MySQL queries are probably to complicated.
Script:
#!/usr/bin/env bash
# DB Details:
db_user="user"
db_pass="password"
db_host="localhost"
# Date
date_stamp=$(date +%d%m%Y)
# Initial Setup
data_dir="/tmp/backup"
# Checks
if [ ! -e /usr/bin/s3cmd ]; then
echo "Required package (http://s3tools.org/s3cmd)"
exit 2
fi
if [ -e /usr/bin/gpg ]; then
gpg_key=$(gpg -K | tr -d "{<,>}" | awk '/an#example.com/ { print $4 }')
if [ "$gpg_key" != "an#example.com" ]; then
echo "No GPG key"
exit 2
fi
else
echo "No GPG package"
exit 2
fi
if [ -d $data_dir ]; then
rm -rf $data_dir/* && chmod 700 $data_dir
else
mkdir $data_dir && chmod 700 $data_dir
fi
# S3 buckets
bucket_1=s3://test/
# Download backup
for backup in $(s3cmd ls s3://test/ | awk '{ print $2 }')
do
latest=$(s3cmd ls $backup | awk '{ print $2 }' | sed -n '$p')
random=$(s3cmd ls $latest | shuf | awk '{ print $4 }' | sed -n '1p')
s3cmd get $random $data_dir >/dev/null 2>&1
done
# Decrypting Files
for file in $(ls -A $data_dir)
do
filename=$(echo $file | sed 's/\.e//')
gpg --out $data_dir/$filename --decrypt $data_dir/$file >/dev/null 2>&1 && rm -f $data_dir/$file
if [ $? -eq 0 ]; then
# Decompressing Files
bzip2 -d $data_dir/$filename
if [ $? -ne 0 ]; then
echo "Decompression Failed!"
fi
else
echo "Decryption Failed!"
exit 2
fi
done
# MySQL Restore
printf "%-40s%-30s%-30s\n\n" Database: Table: Entries:
for dump in $(ls -A $data_dir)
do
mysql -h $db_host -u $db_user -p$db_pass < $data_dir/$dump
if [ $? -eq 0 ]; then
# Random DBs query
db=$(echo $dump | sed 's/\.sql//')
random_table=$(mysql -h $db_host -u $db_user -p$db_pass $db -e "SHOW TABLES" | grep -v 'Tables' | shuf | sed -n '1p')
db_entries=$(mysql -h $db_host -u $db_user -p$db_pass $db -e "SELECT * FROM $random_table" | grep -v 'id' | wc -l)
printf "%-40s%-30s%-30s\n" $db $random_table $db_entries
mysql -h $db_host -u $db_user -p$db_pass -e "DROP DATABASE $db"
else
echo "The system was unable to restore backups!"
rm -rf $data_dir
exit 2
fi
done
#Remove backups
rm -rf $data_dir
You'll get the best answers if you ask specific questions (rather than, "please review my code")...and if you limit each post to a single question. Regarding emailing the output of your printf statements:
You can group statements into a block and then pipe the output of a block into another program. For example:
{
echo "This is a header"
echo
for x in {1..10}; do
echo "This is row $x"
done
} | mail -s "Here is my output" lars#example.com
If you want to make the email subject conditional upon the success or
failure of something elsewhere in the script, you can (a) save your
output to a file, and then (b) email the file after building the
subject line:
{
echo "This is a header"
echo
for x in {1..10}; do
echo "This is row $x"
done
} > output
if is_success; then
subject="SUCCESS: Here is your output"
else
subject="FAILURE: Here are your errors"
fi
mail -s "$subject" lars#example.com < output

psql --(record|field)-separator NUL

Is there some way to make psql separate the fields and records by \0, aka NUL? It's the only way to be able to pass arbitrary data to Bash scripts.
Based on Matthew Wood's answer, I would expect this to print more that 1 on a newly initialized database:
declare -i count=0
echo "\pset recordsep '\000'
\f '\000'
select typname from pg_type" | \
sudo -iu postgres psql --no-align --quiet --tuples-only -d dbname -U username | while IFS= read -r -d ''
do
#echo "$REPLY"
let count++
done
if [ -n "$REPLY" ]
then
#echo "$REPLY"
let count++
fi
echo $count
Workaround: Iff the SELECT results are unique, you can use this workaround to handle one at a time:
next_record() {
psql --no-align --quiet --tuples-only -d dbname -U username <<SQL
SELECT colname
FROM tablename
WHERE colname > '${1}'
ORDER BY colname
LIMIT 1
SQL
}
last_col=
while true
do
colx="$(next_record "$last_col"; printf x)"
if [ "$colx" = x ]
then
exit
fi
col="${colx%$'\nx'}" # The extra \n character is from psql
# Do your thing here
col_escaped="${col//"'"/''}" # Double single quotes
col_escaped="${col_escaped//\\/\\\\}" # Double backslashes
last_col="$col_escaped"
done
This is not supported. psql uses C print functions to print out the result tables, and printing a zero byte just doesn't work there.
Update: This is now supported in PostgreSQL 9.2-to-be (git).
Try this:
psql --field-separator '\000' --no-align -c '<your query>'
Edit: Maybe not. However, it appear to work in psql using these commands:
\f '\000'
\a
Newer versions of psql support the --field-separator-zero flag.

Resources