I have a bash script. I want to run a postgres command with ssh that pipes a local file. The problem is the psql command prompts for a password, and my sql file gets piped into that. How do I write a command that pipes after I type in the password?
ssh server "psql -W -h db_host -p 5432 -U db_user -d postgres" < staging.sql
I suggest to break it down into multiple steps:
# Transfer the sql file to the server
scp staging.sql server
# Excute the queries in that file with psql over ssh
# Notes:
# - ssh -t enforces terminal allocation. You may try it without this option and see if it still works.
# - psql -f FILENAME reads commands from file
#
ssh -t server \
'psql -W -h db_host -U db_user -d postgres -f staging.sql; rm staging.sql'
Related
I'm trying to run a script on a remote server with either password credentials or .pem key access and I'm getting errors no matter which solution I've found etc.
bash script content:
#!/bin/bash
sudo fdisk -l
ssh -T -i "~/.ssh/keys/key.pem" ubuntu#host "sudo bash <(wget -qO- http://host.com/do.sh)"
Error: bash: /dev/fd/63: No such file or director
ssh user#host.com 'echo "password" | sudo bash <(wget -qO- http://www.host.io/do.sh)'
Error: sudo: a password is required
ssh -t user#host.com "echo password | sudo fdisk -l"
Works but still gives me the password propmt
echo -t pass | ssh user#host "sudo bash <(wget -qO- http://host.com/do.sh)"
echo -tt pass | ssh user#host "sudo bash <(wget -qO- http://host.com/do.sh)"
Error: bash: /dev/fd/63: No such file or directory
// And I also get the password prompt
echo -tT pass | ssh user#host "sudo bash <(wget -qO- http://host.com/do.sh)"
Error: sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required
// And I also get the password prompt
// This works but I still get the password propmt
ssh user#host 'echo "password" | sudo -S sudo fdisk -l'
These are different variations of the supposed solutions from other places.
What I'm trying to do:
Is to run a script from a URL on the remote server while echoing the password to the cmd so I don't get propmt to input the password manually.
To be able to do the same thing above with using the .pem key variant also
For an explanation for commands except the first one, You can't do stdin-redirect a password to ssh if ssh requires interactively. ssh only allows manual typing if you use a password.
Your first error said that bash can't read a file descriptor. So ssh via ~/.ssh/keys/key.pem works. To run the shell command on the fly,
ssh -T -i "~/.ssh/keys/key.pem" ubuntu#host "curl -fsSL http://host.com/do.sh | sudo bash"
Does your script really need to run with sudo??
If not, then try this:
ssh user#host "curl -s -o do.sh 'http://host.com/do.sh'; source do.sh"
To set up my project, I have to do quite a few commands and I'm trying to script this away. Some of these are in psql; so normally I'd go
psql -U postgres -h localhost -p 5433
(psql) create database test_database
(psql) \c test_database
(psql) \i integration-test/src/test/resources/init.sql
The .init.sql contains stuff to fill the database with mock data.
In my bash script, I tried reducing this to
psql -U postgres -h localhost -p 5433 -c "create database test_database; \c test_database; \i integration-test/src/test/resources/init.sql"
However, this gets me
ERROR: syntax error at or near "\"
LINE 1: create database test_database; \c fcs_analytics; \i integrat...
^
How do I execute these commands properly from my script?
Have you tried ?
psql -U postgres -h localhost -p 5433 << 'EOF'
create database test_database
\c test_database
\i integration-test/src/test/resources/init.sql
EOF
I have a bash file should bring the postgres docker container online and then run a .sql file to create the databases. But it's throwing the error.
psql: error: provision-db.sql: No such file or directory
I have checked the path and the file exists at the same level of this bash script. Following is the content of my bash file.
#!/usr/bin/env bash
docker-compose up -d db
# Ensure the Postgres server is online and usable
until docker exec -i boohoo.postgres pg_isready --host="${POSTGRES_HOST}" --username="${POSTGRES_USER}"
do
echo "."
sleep 1
done
docker exec -i boohoo.postgres psql -h "${POSTGRES_HOST}" -U "${POSTGRES_USER}" -a -q -f provision-db.sql
And this is the provision-db.sql file.
DROP DATABASE "boo-hoo";
CREATE DATABASE "boo-hoo";
GRANT ALL PRIVILEGES ON DATABASE "boo-hoo" TO postgres;
This is the part of docker-compose.yml
version: '3.3'
services:
db:
container_name: boohoo.postgres
hostname: postgres.boohoo
image: postgres
ports:
- "15432:5432"
environment:
POSTGRES_USER: "postgres"
POSTGRES_PASSWORD: "postgres"
The short version
This works
cat provision-db.sql | docker exec -i boohoo.postgres bash -c 'psql -U ${POSTGRES_USER} -w -a -q -f -'
The long version
multiple things here
1) why does following command not find the provision-db.sql?
docker exec -i boohoo.postgres psql -h "${POSTGRES_HOST}" -U "${POSTGRES_USER}" -a -q -f provision-db.sql
because the provision-db.sql is on your host and not in your container. Therefore, when you execute the psql command inside the container it can not find the file
2) Why didn't my first solution work?
cat provision-db.sql | docker exec -i boohoo.postgres psql -h "${POSTGRES_HOST}" -U "${POSTGRES_USER}" -a -q -f - should do the trick asuming provision-db.sql
That is due to the fact, that the variables ${POSTGRES_USER} and ${POSTGRES_PASSWORD} get evaluated on your host machine and I guess they are not set there. In addition, I forgot to specify the -w flag to avoid the password prompt
3) Why does that work?
cat provision-db.sql | docker exec -i boohoo.postgres bash -c 'psql -U ${POSTGRES_USER} -w -a -q -f -'
Well, let's go through it step by step.
First, we print the content of provision-db.sql, which resides on the host machine to stdout and pipe it to the next command via |.
docker-exec executes a command in the container specified (boohoo.postgres). By specifying the -i flag we allow the stdin from your host to go to stdin in the container <- that's important.
In the container, we execute bash -c which is just a wrapper to avoid evaluating the bash variables on the host. We want the variables from the container and by putting it into single quotes we can do that.
docker-exec boohoo.postgres bash -c "echo $POSTGRES_USER"
evaluates the host env variable named POSTGRES_USER.
docker-exec boohoo.postgres bash -c "echo $POSTGRES_USER"
evaluates the container env variable named POSTGRES_USER.
Next we just have to get our postgres command in order.
psql -U ${POSTGRES_USER} -w -a -q -f -
-U specifies the user
-w does not ask for password
-q do it quietly
-f - process whatever you get from stdin
-f is an option for psql and not for docker exec, and psql is running inside the container, so it can only access the file if it is inside the container as well.
Im trying to make a backup in a folder C:\Users\Marko Petričević\Documents\Radni_sati_Backup\proba where "proba" is the name of backup file.
My command looks like this:
pg_dump -h 192.168.130.240 -p 5433 -U postgres -F c postgres > C:\Users\Marko Petričević\Documents\Radni_sati_Backup\proba
and then i get an error: " pg_dump: too many command-line arguments (first is "Petričević\Documents\Radni_sati_Backup\proba") "
But, when I write a command like:
pg_dump -h 192.168.130.240 -p 5433 -U postgres -F c postgres >C:\radni_sati_backup\radni_sati_proba
Everything works, and I get that "radni_sati_proba" file in the directory as I listed in command.
Why is this happening?
Found out what the problem was:
pg_dump -h 192.168.130.240 -p 5433 -U postgres -F c postgres > C:\Users\Marko Petričević\Documents\Radni_sati_Backup\proba
needs to be like this:
pg_dump -h 192.168.130.240 -p 5433 -U postgres -F c postgres > "C:\Users\Marko Petričević\Documents\Radni_sati_Backup\proba"
Problem was the space in path.
Hi I want to automate my workflow of creating Postgres dump and downloading it. I want to do it from my local machine, for now I figured it out so far in two seperate commands:
sshpass -p "FuckinHardPass" ssh andi#1.2.3.4 "pg_dump -U andi andi_some_db -f /home/andi/PSQL_DUMPS/andi_some_db.sql"
sshpass -p "FuckinHardPass" scp -r andi#1.2.3.4:/home/andi/PSQL_DUMPS/andi_some_db.sql .
how can I join it in one command using pipes etc?
Thanks James Hightower for hint, using your answer I complete it in one command:
sshpass -p "FuckinHardPass" ssh andi#1.2.3.4 "pg_dump -U andi andi_some_db" > andi_some_db.sql
As pg_dump defaults to stdout as it's output file, and ssh displays the command's stdout on it's own stdout, you could do something like:
ssh andi#1.2.3.4 'pg_dump -U andi andi_some_db' > andi_some_db.sql
which would save the output from the command on your local disk as andi_some_db.sql
Depending on the size of your dump and the speed of your connection, you could perhaps benefit from pre-compressing your output:
ssh andi#1.2.3.4 'pg_dump -U andi andi_some_db | gzip' > andi_some_db.sql.gz
And so on.