I can successfully run SQL (Postgres) files from command line following instructions here:
Run a PostgreSQL .sql file using command line arguments
In particular, I use something like
psql -d DBPASSWORD -a -f FILENAME
Problem is that this (and specifically, I believe the -a) prints the sql code out to the terminal. This is annoying because I am running a lot of files in sequence within a Python script using subprocess, and I would rather not have the SQL code print out in terminal. Is there a way to not print the SQL code out to terminal?
EDIT: I've tried adding the -q option like people said, but the code in the SQL file is still being printed out to terminal.
What I tried was
psql -q -d DBPASSWORD -a -f FILENAME
psql -d DBPASSWORD -q -a -f FILENAME
psql -d DBPASSWORD -a -q -f FILENAME
psql -d DBPASSWORD -a -f FILENAME -q
And in each of those cases, the code in FILENAME is being printed to terminal
You may want to redirect STDOUT, STDERR or both to a log file.
Something like one of these
psql ... > out.log
psql ... 2> err.log
psql ... &> out_and_err.log
Related
I have a bash script. I want to run a postgres command with ssh that pipes a local file. The problem is the psql command prompts for a password, and my sql file gets piped into that. How do I write a command that pipes after I type in the password?
ssh server "psql -W -h db_host -p 5432 -U db_user -d postgres" < staging.sql
I suggest to break it down into multiple steps:
# Transfer the sql file to the server
scp staging.sql server
# Excute the queries in that file with psql over ssh
# Notes:
# - ssh -t enforces terminal allocation. You may try it without this option and see if it still works.
# - psql -f FILENAME reads commands from file
#
ssh -t server \
'psql -W -h db_host -U db_user -d postgres -f staging.sql; rm staging.sql'
I have been tasked with replacing ISQL in a lot of our bash scripts with sqlcmd. ISQL allows piping a variable in it's execution.
An example would be:
SQL_STATEMENT="SELECT TOP 1 SYS_USER_NAME FROM SYS_USER"
echo $SQL_STATEMENT | isql -b -d, $DSN $DBUID $DBPWD >> setupdb_test.txt
From what I can tell this is not viable in sqlcmd. How can I do this? What flags does sqlcmd have to allow this to happen?
Here is what I have tried and have had a good result BUT I really do not want to create the file sql_command.sql every time a particular script runs:
echo "SELECT TOP 1 SYS_USER_NAME FROM SYS_USER" > sql_command.sql
sqlcmd -S $DB -U $DBUID -P $DBPWD -d $DSN -i sql_command.sql >> setupdb_test.txt
Programs originating on Windows can be picky about how they handle non-regular files and I don't have the opportunity to test, but you can try the typical Unix tricks for providing a "file" with data from an echo.
Either /dev/stdin:
echo "SELECT TOP 1 SYS_USER_NAME FROM SYS_USER" | sqlcmd -S "$DB" -U "$DBUID" -P "$DBPWD" -d "$DSN" -i /dev/stdin
or process substitution:
sqlcmd -S "$DB" -U "$DBUID" -P "$DBPWD" -d "$DSN" -i <(echo "SELECT TOP 1 SYS_USER_NAME FROM SYS_USER")
I'm trying to run a script (automation.sh) automated in crontab.
(I'm on Ubuntu 14.04).
#!/usr/bin/env bash
day=$(date +%F -d'yesterday')
cat /home/tomi/logs/$day |grep registration > /home/tomi/registrations/$day
cat /home/tomi/logs/$day |grep free_tree > /home/tomi/free_tree/$day
cat /home/tomi/logs/$day |grep super_tree > /home/tomi/super_tree/$day
psql -U hello -d postgres -c "\COPY registrations FROM '/home/tomi/registrations/$day' DELIMITER ' '";
psql -U hello -d postgres -c "\COPY free_tree FROM '/home/tomi/free_tree/$day' DELIMITER ' '";
psql -U hello -d postgres -c "\COPY super_tree FROM '/home/tomi/super_tree/$day' DELIMITER ' '";
psql -U hello -d postgres -f daily_active_users.sql > /home/tomi/tmp1
psql -U hello -d postgres -f daily_revenue.sql > /home/tomi/tmp2
If I run this script normally from the command line, then the last two lines generate tmp1 and tmp2 with data in them. (That's the expected result.)
However, if I run this very same script in crontab, everything works, but the last two lines generate empty files (tmp1 and tmp2).
The tricky thing is that when I break this script into two scripts (eg. automated.sh and automated2.sh) and run the last two lines in crontab 5 minutes later (via this automated2.sh script), tmp1 and tmp2 are generated correctly, with data in them.
Any idea, what can cause this?
Answer is in the comments:
full path is missing on last two lines in automation.sh!
Thanks!
What should I do for making it work?
#!/bin/bash
TABLENAMES="user_stats"
ssh -t railsapps#xxx.xxx.xxx.xx -p xxx bash -c "'
for TABLENAME in $TABLENAMES
do
psql -d mydb -P format=unaligned -P tuples_only -P fieldsep=\, -c "SELECT * FROM $TABLENAME" > /tmp/$TABLENAME
done
'"
General problem: how to periodically dump the database tables to a local machine from a psql database in a single bash script run on Mac OS X?
Firstly, you should test your SQL and bash scripts remotely (do SSH interactively).
I think your problem is caused by a bad mix of quote / double-quote. I think the star (*) and $TABLENAME are expensed before the SSH call, so too early. Try to put a backslash before the $ sign.
You should use the verbose or the debug option, to help to understand what is really executed:
ssh -t railsapps#xxx.xxx.xxx.xx -p xxx bash -vxc "'
for TABLENAME in \$TABLENAMES; do
psql -d mydb -P format=unaligned -P tuples_only -P fieldsep=\, -c "SELECT \* FROM \$TABLENAME" > /tmp/\$TABLENAME
done
'"
I have set the PATH environment and then run psql on command prompt and then whatever command I write nothing happens, it doesn't throw any error message too.
even very basic commands as;
psql -l
pg_dump -U {user-name} {source_db} -f {dumpfilename.sql}
Do you have any idea?
You have to finish each command with a semicolon (;). Until you do so, psql thinks you're still writing the same (multi-line) command.
Also, psql -l and pg_dump -U {user-name} {source_db} -f {dumpfilename.sql} are not psql commands. These should be ran from your shell (without the semicolons), not from the psql terminal.