Is anyone doing versioning of Greenplum Code?
Can anyone help on this, is there any direct tool or method which can help in manage version of GP DDL/Code of database objects (Tables, Functions, Views)
Regards,
I've had the best success of using a cron job to commit any changes to a source control system like git or subversion on a daily basis.
--create a SQL file per schema
for i in $(psql -t -A -c "select nspname from pg_namespace where nspname not like 'pg_%' and nspname not like '%toolkit' and nspname not in ('information_schema', 'madlib', 'public') order by nspname;"); do
echo $i
pg_dump -s -n $i -f $i.sql
done
for i in $(ls *.sql); do
#your code to commit the changes to source control like git
done
Related
Apologies if the title isn't worded very well, hard to explain exactly what I'm trying to do without an example
I am running a database backup command that creates a file with a timestamp. In the same command I am then uploading that file to a remote location.
pg_dump -U postgres -W -F t db > $backup_dir/db_backup_$(date +%Y-%m-%d-%H.%M.%S).tar && gsutil cp $backup_dir/db_backup_$(date +%Y-%m-%d-%H.%M.%S).tar $bucket_dir
As you can see here it is creating the timestamp during the pg_dump command. However in the 2nd half of the command, the timestamp will now be different and it won't find the file.
I'm looking for a way to 'save' or assign the value of the backup file name from the first half of the command, so that I can then use it in the 2nd half of the command.
Ideally this would be done across 2 separate commands however in this particular use case I'm limited to 1.
a variation of the advice already given in comments -
fn=db_backup_$(date +%Y-%m-%d-%H.%M.%S).tar &&
pg_dump -U postgres -W -F t db > "$backup_dir/$fn" &&
gsutil cp "$backup_dir/$fn" "$bucket_dir"
The $fn var makes the whole thing shorter and more readable, too.
I have read all other solutions and none adapts to my needs, I do not use Java, I do not have super user rights and I do not have API's installed in my server.
I have select rights on a remote PostgreSQL server and I want to run a query in it remotely and export its results into a .csv file in my local server.
Once I manage to establish the connection to the server I first have to define the DB, then the schema and then the table, fact that makes the following lines of code not work:
\copy schema.products TO '/home/localfolder/products.csv' CSV DELIMITER ','
copy (Select * From schema.products) To '/home/localfolder/products.csv' With CSV;
I have also tried the following bash command:
psql -d DB -c "select * from schema.products;" > /home/localfolder/products.csv
and logging it with the following result:
-bash: home/localfolder/products.csv: No such file or directory
I would really appreciate if someone can show a light on this.
Have you tried this? I do not have psql right now to test it.
echo “COPY (SELECT * from schema.products) TO STDOUT with CSV HEADER” | psql -o '/home/localfolder/products.csv'
Details:
-o filename Put all output into file filename. The path must be writable by the client.
echo builtin + piping (|) pass command to psql
Aftr a while a good colleague deviced this solution which worked perfectly for my needs, hope this can help someone.
'ssh -l user [remote host] -p [port] \'psql -c "copy (select * from schema.table_name') to STDOUT csv header" -d DB\' > /home/localfolder/products.csv'
Very similar to idobr's answer.
From http://www.postgresql.org/docs/current/static/sql-copy.html:
Files named in a COPY command are read or written directly by the server, not by the client application.
So, you'll always want to use psql's \copy meta command.
The following should do the trick:
\copy (SELECT * FROM schema.products) to 'products.csv' with csv
If the above doesn't work, we'll need an error/warning message to work with.
You mentioned that the server is remote, however you are connecting to a localhost. Add the -h [server here] or set the ENV variable
export PGHOST='[server here]'
The database name should be the last argument, and not with -d.
And finally that command should have not failed, my guess is that that directory does not exist. Either create it or try writing to tmp.
I would ask you to try the following command:
psql -h [server here] -c "copy (select * from schema.products) to STDOUT csv header" DB > /tmp/products.csv
I'm looking for a way to restore my DB from a prior backup. However, the backup should not simply overwrite all existing records but instead add only the difference between current DB and the backup file. If no "non existent" records are stored in the backup, nothing should happen. The backups were made with mysqldump. Any clues?
Thanks in advance
Here is a less manual answer:
mysqldump -t --insert-ignore --skip-opt -u USER -pPASSWORD -h 127.0.0.1 database > database.sql
That export command with the -t --insert-ignore --skip-opt options will give you a sql dump file with no DROP TABLE or CREATE TABLE commands and every INSERT is now an INSERT IGNORE.
BONUS:
This will dump a single table in the same way:
mysqldump -t --insert-ignore --skip-opt -u USER -pPASSWORD -h 127.0.0.1 database table_name > table_name.sql
I needed this today and could not help but to share it!
Remove the DROP TABLE and CREATE TABLE statements from the dump file. Change the INSERT statements to INSERT IGNORE. Then load the backup file and it should not update any duplicate rows.
How do I check if a database exists in bash?
I'm planning to make an automated backup script, where each website's db name is:
(siteName) mysite.com = mysite_com (dbName)
You can check if a file exists with the -e bash command. MongoDB creates a Namespace file for each database, so search for those. Something like
if [ -e /data/db/$DBNAME.ns ]; then
`mongodump -d $DBNAME --out /backup/directory`
fi
Assuming your data is stored in /data/db
Have a look at the mongodump command which can be used to backup data from your mongo db instance
http://www.mongodb.org/display/DOCS/Backups
I am trying to automate a set of procedures that create TEMPLATE databases.
I have a set of files (file1, file2, ... fileN), each of which contains a set of pgsql commands required for creating a TEMPLATE database.
The contents of the file (createdbtemplate1.sql) looks roughly like this:
CREATE DATABASE mytemplate1 WITH ENCODING 'UTF8';
\c mytemplate1
CREATE TABLE first_table (
--- fields here ..
);
-- Add C language extension + functions
\i db_funcs.sql
I want to be able to write a shell script that will execute the commands in the file, so that I can write a script like this:
# run commands to create TEMPLATE db mytemplate1
# ./groksqlcommands.sh createdbtemplate1.sql
for dbname in foo foofoo foobar barbar
do
# Need to simply create a database based on an existing template in this script
psql CREATE DATABASE $dbname TEMPLATE mytemplate1
done
Any suggestions on how to do this? (As you may have guessed, I'm a shell scripting newbie.)
Edit
To clarify the question further, I want to know:
How to write groksqlcommands.sh (a bash script that will run a set of pgsql cmds from file)
How to create a database based on an existing template at the command line
First off, do not mix psql meta-commands and SQL commands. These are separate sets of commands. There are tricks to combine those (using the psql meta-commands \o and \\ and piping strings to psql in the shell), but that gets confusing quickly.
Make your files contain only SQL commands.
Do not include the CREATE DATABASE statement in the SQL files. Create the db separately, you have multiple files you want to execute in the same template db.
Assuming you are operating as OS user postgres and use the DB role postgres as (default) Postgres superuser, all databases are in the same DB cluster on the default port 5432 and the role postgres has password-less access due to an IDENT setting in pg_hba.conf - a default setup.
psql postgres -c "CREATE DATABASE mytemplate1 WITH ENCODING 'UTF8'
TEMPLATE template0"
I based the new template database on the default system template database template0. Basics in the manual here.
Your questions
How to (...) run a set of pgsql cmds from file
Try:
psql mytemplate1 -f file
Example script file for batch of files in a directory:
#! /bin/sh
for file in /path/to/files/*; do
psql mytemplate1 -f "$file"
done
The command option -f makes psql execute SQL commands in a file.
How to create a database based on an existing template at the command line
psql -c 'CREATE DATABASE my_db TEMPLATE mytemplate1'
The command option -c makes psql execute a single SQL command string. Can be multiple commands, terminated by ; - will be executed in one transaction and only the result of the last command returned.
Read about psql command options in the manual.
If you don't provide a database to connect to, psql will connect to the default maintenance database named "postgres". In the second answer it is irrelevant which database we connect to.
you can echo your commands to the psql input:
for dbname in foo foofoo foobar barbar
do
echo """
CREATE DATABASE $dbname TEMPLATE mytemplate1
""" | psql
done
If you're willing to go the extra mile, you'll probably have more success with sqlalchemy. It'll allow you to build scripts with python instead of bash, which is easier and has better control.
As requested in the comments: https://github.com/srathbun/sqlCmd
Store your sql scripts under a root dir
Use dev,tst,prd parametrized dbs
Use find to run all your pgsql scripts as shown here
Exit on errors
Or just git clone the whole tool from here
For that use case where you have to do it....
Here is a script I've used for importing JSON into PostgreSQL (WSL Ubuntu), which basically requires that you mix psql meta commands and SQL in the same command line. Note use of the somewhat obscure script command, which allocates a pseudo-tty:
$ more update.sh
#!/bin/bash
wget <filename>.json
echo '\set content `cat $(ls -t <redacted>.json.* | head -1)` \\ delete from <rable>; insert into <table> values(:'"'content'); refresh materialized view <view>; " | PGPASSWORD=<passwd> psql -h <host> -U <user> -d <database>
$