In psql I'd like to exec a shell command with interpolated psql variables like
postgres=> \! pg_dump -Fc -d :source_db | pg_restore -d :dest_db
But the psql docs make clear that this doesn't work:
the entire remainder of the line is always taken to be the argument(s) of !, and neither variable interpolation nor backquote expansion are performed in the arguments.
I've found the following workarounds, but they feel unnecessarily cumbersome. Is there a better way? Maybe there are feature requests for a meta-command like \! that does interpolation?
Workaround 1: build command with SQL and use \g to pass it to bash
postgres=> SELECT 'pg_dump -Fc -d ' || :'source_db' || ' | pg_restore -d ' || :'dest_db' \g (tuples_only=on) | bash
Workaround 2: move values to environment variables with \setenv
postgres=> \setenv SOURCE_DB :source_db
postgres=> \setenv DEST_DB :dest_db
postgres=> \! pg_dump -Fc -d $SOURCE_DB | pg_restore -d $DEST_DB
Related
I want to assign the output of a mongo command (i.e database names) to a bash array variable but I am not sure how to go about it.
I am getting an error :
dump.sh
Starting-BACKUP
dump.sh
./dump.sh: line 16: declare: `–a': not a valid identifier
./dump.sh: line 24: mongo: command not found
Database backup was successful
when I attempt using dump.sh below:
#!/bin/bash
declare –a databases=()
databases=$(mongo --quiet --uri="mongodb://root:mypassword#mongodb-prom/admin" --authenticationDatabase admin --eval="show dbs;" | cut -d " " --field 1)
echo $databases
Ordinarily I am able to get a listing of the databases when I kubectl into the pod with following steps :
$ kubectl exec -it mongodb-prom-xxxxxxxxx-dddx4 -- sh
$ mongo
> use admin
> db.auth('root','mypassword')
> show dbs
admin 0.000GB
platforms 0.000GB
users 0.000GB
I am not sure why the mongo command is not being recognized because the same script is able to execute the mongodump command below :
mongodump --uri="<uri_here>" --authenticationDatabase admin --gzip --archive=/tmp/"<variable_here>".gz
UPDATE : This is the associated Dockerfile. My expectation is that both mongo and mongodump should be working by default in a mongo container but it seems only mongodump is working for now
FROM mongo
WORKDIR /opt/backup/
WORKDIR /usr/src/configs
COPY dump.sh .
RUN chmod +x dump.sh
My two issues :
Is my syntax correct for the variable assignment (I suspect its not correct) ?
How should I properly declare the array variable to avoid the warnings ?
NB : Mongo tooling is already installed on the container and is actually working for mongodump
You don't need declare -a. Simply putting the value inside () in the assignment will make it an array.
To get all the elements of an array, you have to use ${variable[#]}. $variable is equivalent to ${variable[0]} and just retrieves the first element.
databases=($(mongo --quiet --uri="mongodb://root:mypassword#mongodb-prom/admin" --authenticationDatabase admin --eval="show dbs;" | cut -d " " --field 1))
echo "${databases[#]}"
With bashv4+, mapfile is available, with Process Substitution.
mapfile -t databases < <(mongo --quiet --uri="mongodb://root:mypassword#mongodb-prom/admin" --authenticationDatabase admin --eval="show dbs;" | cut -d " " --field 1)
Now check the value of the databases array
declare -p databases
Your code has
declare -a databases=()
but
databases=$(...)
is not an array, the $(...) construct is Command Substitution.
See also Bash Arrays
I have been tasked with replacing ISQL in a lot of our bash scripts with sqlcmd. ISQL allows piping a variable in it's execution.
An example would be:
SQL_STATEMENT="SELECT TOP 1 SYS_USER_NAME FROM SYS_USER"
echo $SQL_STATEMENT | isql -b -d, $DSN $DBUID $DBPWD >> setupdb_test.txt
From what I can tell this is not viable in sqlcmd. How can I do this? What flags does sqlcmd have to allow this to happen?
Here is what I have tried and have had a good result BUT I really do not want to create the file sql_command.sql every time a particular script runs:
echo "SELECT TOP 1 SYS_USER_NAME FROM SYS_USER" > sql_command.sql
sqlcmd -S $DB -U $DBUID -P $DBPWD -d $DSN -i sql_command.sql >> setupdb_test.txt
Programs originating on Windows can be picky about how they handle non-regular files and I don't have the opportunity to test, but you can try the typical Unix tricks for providing a "file" with data from an echo.
Either /dev/stdin:
echo "SELECT TOP 1 SYS_USER_NAME FROM SYS_USER" | sqlcmd -S "$DB" -U "$DBUID" -P "$DBPWD" -d "$DSN" -i /dev/stdin
or process substitution:
sqlcmd -S "$DB" -U "$DBUID" -P "$DBPWD" -d "$DSN" -i <(echo "SELECT TOP 1 SYS_USER_NAME FROM SYS_USER")
I want to export a random entry of my database into a file with the command
SELECT * FROM my_table ORDER BY RANDOM() LIMIT 1 \g /path/file;
This query works if I enter it in my db terminal, but I want to us this query with a bash script but then I get the error: syntax error at or near "\g"
My bash script looks like this:
PGPASSWORD=*** psql -U user -d db_name -h localhost -p port -t -c "SELECT * FROM my_table ORDER BY RANDOM() LIMIT 1 \g /path/file"
Bash is interpreting the string and trying to interpolate it. Probably, escaping the backslash will solve your problem.
PGPASSWORD=*** psql -U user -d db_name -h localhost -p port -t -c "SELECT * FROM my_table ORDER BY RANDOM() LIMIT 1 \\g /path/file"
A SQL statement terminated by \g is not supported by the -c command switch. Per documentation of -c:
-c command
...
command must be either a command string that is completely parsable by the server (i.e., it contains no psql-specific features), or a single backslash command. Thus you cannot mix SQL
and psql meta-commands with this option
To redirect the results to a file, there are several options:
shell redirection: psql [other options] -Atc 'SELECT...' >/path/to/data.txt
-A is to switch to unaligned mode (no space fillers to align columns).
put the SQL part in a heredoc text instead of the command line:
psql [options] <<EOF
SELECT ... \g /path/to/file
EOF
This form has the advantage that multiline statements or multiple statements are supported directly.
\copy of the query. Be aware that COPY to a FILE is different: it creates the file on the server with the permissions of postgres and requires being a database superuser. COPY TO STDOUT works too but is not better than SELECT concerning the redirection.
I found a solution for my script, and now it works.
#!/bin/bash
RANDOM_NUMBER=0
while true
do
for i in `seq 1`
do
RANDOM_NUMBER=$(($RANDOM % 100000))
echo $RANDOM_NUMBER
PGPASSWORD=*** psql -U user_name -d db_name -h localhost -p PORT -c
"INSERT INTO numbers (number) VALUES ('$RANDOM_NUMBER');"
done
sleep 10
for i in `seq 1`
do
PGPASSWORD=*** psql -U user_name -d db_name -h localhost -p PORT -c
"DELETE FROM numbers WHERE id = (SELECT id FROM numbers ORDER BY RANDOM() LIMIT 1);"
done
done
I'm working on a bash script used to install an application on a server and create a postgresql user for the application.
The matter is that if the user contains a special char, it's not escaped. So I would like to escape the name in the query.
Here is a part of my code:
db_user="test-test"
db_password=1234
su - postgres -c "psql -c \"CREATE USER $db_user WITH PASSWORD '$db_password';\""
So I have to escape:
the command for su -c ...
the query for psql -c ...
the user name in $db_user
the password in $db_password
Of course, if I add quotes around $db_user it's not working anymore. I tried others solutions, but none working.
Any idea?
Thanks
Let the shell do the work for you for you. In this case, that means printf '%q' -- which returns an eval-safe version of each literal argument.
cmd=( psql -c "CREATE USER $db_user WITH PASSWORD '$db_password';" )
printf -v cmd_q '%q ' "${cmd[#]}"
su - postgres -c "$cmd_q"
By the way, substituting literal text into SQL queries this way is massively insecure -- but if you follow the process above you're prone to only SQL injection bugs, not shell injection bugs.
To avoid SQL injection bugs as well, you want to avoid textual substitution of values into your queries at all.
# This is inefficient, but makes it completely unambiguous that our heredocs are literal
query=$(cat <<'EOF'
CREATE USER :new_user_name WITH PASSWORD :'new_user_password';
EOF
)
cmd=(
psql \
--set=new_user_name="$db_user" \
--set=new_user_password="$db_password"
-c "$query"
)
printf -v cmd_q '%q ' "${cmd[#]}"
su - postgres -c "$cmd_q"
See http://caryrobbins.com/dev/postgres-scripting/ for more on this technique. The classic XKCD linked in its header (also useful as a concrete example of the risks the original code was taking!) has its original source at https://xkcd.com/327/.
I'm getting 7: Syntax error: "(" unexpected error while running bellow code on Ubuntu. But It's run on centos without any issues.
#!/bin/sh
#
TODATE=`date '+%Y-%b-%d'`
#
# Backup Creation for Databases
#
databases=(`echo 'show databases;' | mysql -u root -ppaSSword | grep -v ^Database$`)
for DB in "${databases[#]}"; do
mysqldump --force --opt --user=root --password=paSSword $DB | gzip > /mnt/Backup/DB/${DB}_${TODATE}.sql.gz
done
#
Please help me to solve this.
I can't figure out problem. But,
I'm using bellow code for backup. It's working fine with Ubuntu
#!/bin/bash
#
TODATE=`date '+%Y-%b-%d'`
databases="$(mysql -u root -ppaSSword -Bse 'show databases')"
for DB in $databases
do
mysqldump -u root -psqlMYadmin $DB | gzip > /mnt/Backup/DB/${DB}_${TODATE}.sql.gz
done
You can redirect the 'show databases' output to dump.txt file, if done then try.
#!/bin/bash
da=$(date +"%d-%m-%y")
for db in `cat dump.txt` ; do mysqldump --force --opt --user=root --password=paSSword $db | gzip /path/to/backup/$db_"$da".sql.gz ; done
You need to escape the last '$' on the line databases= ...
There's only one ( in the script and you have the shebang line #!/bin/sh. My best guess is that the program /bin/sh does not recognize array assignment, whereas /bin/bash would.
Change your shebang to #!/bin/bash.
You'd probably do better to use $(...) in place of the back ticks.) Also, as Sami Laine points out in his answer, it would be better if you quoted the regex to the grep command (though it is not the cause of your problem):
databases=( $(echo 'show databases;' | mysql -u root -ppaSSword | grep -v '^Database$') )