I have a function that reads a list of names and uses that as input into another command like so:
runMain() {
getName=$(PGPASSWORD=$_clpw1 psql -h myendpoint.com -U ops_readonly -d dev -p 5439 -t -c "select datname from pg_database where datname not like 'template%' and datname not like 'tealium%' and datname not like 'padb%' and datname not like 'services%' and datname not like 'sales%' and datname not like 'dev%' and datname not like 'demo_%' and datname not like '%_demo' and datname not like 'nt_%';")
echo "${getName}" >> "${_file}"
for db in $(cat "${_file}");
do
getSchema=$(PGPASSWORD=$_clpw2 psql -h myendpoint.com -U masteruser -d "${db}" -p 5439 -t -c "select distinct 'GRANT SELECT ON ALL TABLES IN SCHEMA ' || table_schema ||' TO ops_readonly;' FROM information_schema.tables where table_catalog='%${db}%' and table_schema not in ('pg_catalog','information_schema');")
echo "${getSchema}" >> "${_script}"
done
}
I do an echo "${getSchema}" >> "${_script}" so I can output my query findings to a file.
I can see the output file has been touched and its like 32k, however the file its empty.
I've tested the command line string separately and it works and i'm getting the output I expect.
Is there a better way to capture the output to a file? What am I missing? Thank you.
Your file is empty (filled with 32k of blank lines) because ${getName} and ${getSchema} are both empty. Therefore, the problem is in those psql commands.
It's okay to put one level of double quotes inside a "$(…)" (never mind that SO's highlighting messes it up), so you can do this:
runMain() {
getName="$(PGPASSWORD="$_clpw1" psql -h myendpoint.com -U ops_readonly -d dev -p 5439 -t -c "select datname from pg_database where datname not like 'template%' and datname not like 'tealium%' and datname not like 'padb%' and datname not like 'services%' and datname not like 'sales%' and datname not like 'dev%' and datname not like 'demo_%' and datname not like '%_demo' and datname not like 'nt_%';" 2>&1)"
echo "${getName}" >> "${_file}"
for db in $(grep -o '[[:alnum:].-][[:alnum:].-]*' "${_file}");
do
getSchema="$(PGPASSWORD="$_clpw2" psql -h myendpoint.com -U masteruser -d "${db}" -p 5439 -t -c "select distinct 'GRANT SELECT ON ALL TABLES IN SCHEMA ' || table_schema ||' TO ops_readonly;' FROM information_schema.tables where table_catalog='%${db}%' and table_schema not in ('pg_catalog','information_schema');" 2>&1)"
echo "${getSchema}" >> "${_script}"
done
}
In addition to quoting your command substitution, I also quoted your passwords (which matters tremendously if you have spaces or other non-word characters in them!). Finally, I gave your commands 2>&1 to convert standard error into standard output so it can be captured in your output files, which might reveal other problems (like a bad password or bad connection).
The grep command makes the for loop safe to execute, ensuring the inputs are restricted to one or more alphanumeric characters, dots, and hyphens.
An output file was created. I was just getting 0 results. The root of my issue was in the plsql query. Running it via command line, my use of '%${db}%' is valid if running queries for postgres using like and since I am not using like, removing the surrounding % to '${db}' produced my output.
Related
I need in a bash script a IF condition on the existence of a role in a PostgreSQL database. I have found solutions in SQL code [1, 2], but I need something I can use directly in bash, I assume with the help of psql. In [2] there are also psql solutions, but I don't manage to adapt it in a IF statement.
I have tried this unsuccessfully (I am a PostgreSQL and bash newbie):
psql_USER=my
if [ "$( psql -h db -U postgres --no-psqlrc --single-transaction --pset=pager=off --tuples-only --set=ON_ERROR_STOP=1 -tc "SELECT 1 FROM pg_user WHERE usename = $psql_USER" | grep -q 1 )" == '1' ] > /dev/null 2> /dev/null; then
echo "HOURRA !"
fi;
Result is:
Password for user postgres:
ERROR: column « my » does not exist
LINE 1: SELECT 1 FROM pg_user WHERE usename = my
^
I would avoid the quoting problem like this:
if psql -Atq -c "SELECT '#' || usename || '#' FROM pg_user" | grep -q '#'"$psql_USER"'#'
then
echo yes
fi
The psql invocation selects a list of all usernames, prefixed and suffixed with #. The grep has return code 0 if psql_USER contains one of these user names, else 1. The then branch of if is only taken if the return code of the pipeline is 0, that is, if the user exists in the database.
I want to pass argument to a BigQuery script in shell, here is the example of script I wrote
#!/bin/bash
bq query --use_legacy_sql=false --destination_table=abc --append 'select * from `xyz.INFORMATION_SCHEMA.VIEWS` union all Select * from `def.VIEWS`) where table_name = "$1"'
when I run this script and pass the argument, I do not get any errors but no row is appended to the table. whereas when i specify the table_name as rty that row is appended to the table. What am I missing here?
When you run the script you'll get a prompt like:
Waiting on <BIGQUERY_JOB_ID> ... (0s) Current status: DONE
You can inspect the job in many ways, including the bqtool:
bq show -j --format=prettyjson <BIGQUERY_JOB_ID>
If you have jq installed (sudo apt install jq) you can get just the translated query with:
bq show -j --format=prettyjson <BIGQUERY_JOB_ID> | jq '.configuration.query.query'
which will get you something similar to:
select * from xyz.INFORMATION_SCHEMA.VIEWS where table_name = \"$1\"
As you can see the variable is not correctly escaped so no table matches the WHERE filter. To avoid this you can enclose the query in double quotes and the variable in single ones like this:
#!/bin/bash
bq query \
--use_legacy_sql=false \
--destination_table=xyz.abc \
--append \
"select * from xyz.INFORMATION_SCHEMA.VIEWS where table_name='$1'"
You can get the INFORMATION_SCHEMA.VIEWS: command not found error if using back-ticks. You can omit or escape them with a backslash:
"select * from \`xyz\`.INFORMATION_SCHEMA.VIEWS where table_name='$1'"
I'm trying to run the following code but is encountering the "was unexpected at this time" error.
(echo COPY (SELECT ta.colA as name, ta.colB as user_e, ta.colC as user_n, ta.activation_dt, ta.creation_dt, MAX(tb.update_dt) as updated_at, MAX(tb.login_dt) as lastest_login, tc.colD as roleFROM tblA ta, tblB tb, tblC tc WHERE ta.id = tb.tb_id AND ta.tc_id = tc.id AND tc.colD <> 'Guest' GROUP BY ta.colA, ta.colB, ta.colC, ta.activation_dt, ta.creation_dt, tc.colD ORDER BY ta.colA, tc.colD^^^) TO 'E:\Details.csv' CSV DELIMITER ',' HEADER;) | psql -h localhost -p 8060 -U uname -d dbase
Looking for some insights please. Thank you.
Screenshot of error encountered
Try adding some quotes around the SQL, and lose the brackets:
echo "COPY ..." | psql -h localhost -p 8060 -U uname -d dbase
or use -c option:
psql -h localhost -p 8060 -U uname -d dbase -c "COPY ..."
I prefer the -c because it works on all OS
I'm executing the query as follows:
ssh user#XX.XX.1XX.XX "PGPASSWORD=myPassword psql -U psqlUser -h XX.XX.XX.XX -p 5432 -d myDB -c
'INSERT INTO table(\"CPU_IDLE_TIME\",\"TOTAL_SIZE\",\"USED_SIZE\",\"USED_STORAGE_P\") VALUES ($idlecputime,$totalSize,$usedSize,$usedStoragePercentage)';"
I obtain the values previous to this query doing snmpwalks. In order for the query to work the values have to be surrounded by single quotes (' '). I tried putting single quotes around the variable but everytime I get an error because the query is already surroundes by " ' ' ". I can't seem to find the configuration of quotes, or scaping quotes to make it work.
The variables are of type var char, integer and float.
One of the errores I get:
ERROR: syntax error at or near ","
Thanks in advance for your help.
Use printf to format the string for you
ssh user#XX.XX.1XX.XX "PGPASSWORD=myPassword psql -U psqlUser -h XX.XX.XX.XX -p 5432 -d myDB -c ""$(printf 'INSERT INTO table("CPU_IDLE_TIME","TOTAL_SIZE","USED_SIZE","USED_STORAGE_P") VALUES (%d, %d, %d, %d);' $idlecputime $totalSize $usedSize $usedStoragePercentage)"
The $( ) construct executes printf in a subshell.
I have a bash function where I check if a PostgreSQL database already exists.
I capture the output. If database exist PostgreSQL returns the database name as response.
function is_database() {
local database=$1
local output=$(sudo -u postgres psql -c "SELECT datname FROM pg_catalog.pg_database WHERE datname=\"$database\";")
if [[ $output = *"${1}"* ]]
then
return 0
else
return 1
fi
}
is_database test
I get the following error:
column "test" does not exist
I am not searching for a table, but a database.
Use single quotes for string literals:
sudo -u postgres psql \
-c "SELECT datname FROM pg_catalog.pg_database WHERE datname='$database'"
Your code as it is won't work for database names like has spaces or has'quotes.