I have a local db full of data that i want to push to the Heroku's DB in order to populae it.
What is the best way/tools o realize this ?
Thank you!
You could write a script that ports your current DB into a seed file, then run the seed file using heroku run rake db:seed
First dump your local database to a dump file:
PGPASSWORD=mypassword pg_dump -Fc --no-acl --no-owner -h localhost -U myuser mydb > mydb.dump
Replace myuser, mypassword, and mydb with your username, password and database name. If there is no password set, leave out the PGPASSWORD=mypassword part.
Next, you must place the mydb.dump file in a publicly accessible location, so upload it to an FTP server or Amazon S3 bucket (for example).
Then on your local machine run:
heroku pg:backups restore 'https://s3.amazonaws.com/me/mydb.dump' HEROKU_POSTGRESQL_COLOR_URL -a appname
Replace HEROKU_POSTGRESQL_COLOR_URL with the URL for your app's database. If you don't know the URL, you can find it with heroku config | grep HEROKU_POSTGRES. Replace https://s3.amazonaws.com/me/mydb.dump with the URL where you uploaded the dump file. Replace appname with name of your app as defined in Heroku.
Related
I am trying to copy a local PostgreSQL database to Heroku per this article.
Here is what I have done:
1. Make a dump file
pg_dump -Fc --no-acl --no-owner -h localhost -U postgres mydb > mydb.dump
2.Upload dump file to aws my-bucket-name/db-backup folder.
aws s3 cp mydb.dump s3://my-bucket-name/db-backup/mydb.dump
3. Generate a signed URL:
aws s3 presign s3://my-bucket-name/db-backup/mydb.dump --region us-east-2
4. Verify that the signed URL is accessible.
Navigate to the presigned URL in an incognito tab of a browser. It works.
5. Back up to Heroku using the generated signed URL
I am using double quotes around GENERATED_URL because I'm on Windows:
heroku pg:backups:restore --app my-app-name --confirm my-app-name "GENERATED_URL"
For example:
heroku pg:backups:restore --app my-app-name --confirm my-app-name "https://s3.us-east-2.amazonaws.com/s3.console.aws.amazon.com/s3/buckets/my-bucket-name/db-backup/mydb.dump?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIABCDVKE2GXCY3YXL7V%2F20200934%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20200924T164718Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=fb2f51c0d7fbe1234e3740cf23c37f003575d968a1e4961684a47ac627fbae2e"
THE RESULT
I get the following errors:
Restoring... !
! An error occurred and the backup did not finish.
!
! Could not initialize transfer
!
! Run heroku pg:backups:info r021 for more details.
'X-Amz-Credential' is not recognized as an internal or external command,
operable program or batch file.
'X-Amz-Date' is not recognized as an internal or external command,
operable program or batch file.
'X-Amz-Expires' is not recognized as an internal or external command,
operable program or batch file.
'X-Amz-SignedHeaders' is not recognized as an internal or external command,
operable program or batch file.
'X-Amz-Signature' is not recognized as an internal or external command,
operable program or batch file.
I've found others with similar problems, but no solutions. Thanks in advance to anyone who can help.
This is resolved. There were two issues.
PowerShell wasn't properly escaping characters. So, I switched to CMD.
The dump file was invalid.
This line of code produced an invalid dump file:
pg_dump -Fc --no-acl --no-owner -h localhost -U postgres mydb > mydb.dump
Instead, I needed to use the following syntax:
pg_dump -Fc --no-acl --no-owner -h localhost -U postgres -d mydb -f mydb.dump
After making that change, all worked smoothly.
For what it's worth, I had the same issue and my solution was to copy the S3 URL which is formatted as https://s3.amazonaws.com/<bucket_name>/<dump_file>.dump. For some reason the pre-signed URL approach did not work but the public URL did.
I want to be able to clone the contents of our postgre production database to an ownerless local database efficiently. I've successfully done this, but it was a laborious process with the following steps
$ pg_dump [prod_db] > tempfile
[Go through tempfile manually removing all 60ish references to the owner, named 'postgres']
$ cat tempfile > psql [local_db]
Otherwise when I ran the last step, I got a bunch of SQL error messages saying ERROR: role "postgres" does not exist. I tried recreating the local db with a matching 'postgres' owner, but a) I still got the same type of errors, and b) I don't want to have an owner set for my local database if it means I'll have to log into it.
Is there a best practice/efficient way of doing this if I want to re-clone it in future?
Use the -O switch to not have an owner defined in the dump.
Not having an owner set is not normal Postgres design. To avoid having to login to your postgres database you can setup a .pgpass file. This is a plain text file and should be set with 600 permissions. The contents will look like:
hostname:port:database:username:password
Each connection can be on one line.
Additionally, for local db connections, assuming you have setup the rest of your security properly (ssh certs, etc) you can edit your pg_hba.conf file and set local connection authentication method to "trust." This is obviously not recommended for production or sensitive data. The line would look like this:
local all all trust
The default method is "peer". Unless you set your username in a .pgpass file you will still need to connect with psql -U postgres but you will not need to enter a password.
I'm setting up a development environment on heroku for my app and I'm having an issue copying over the DB. My current DB is ClearDB and I usually connect to it via Workbench. However, if I try to export the DB and iimport into my staging environment I get a credential issue.
I found this post on SO with regards to this issue:
Moving/copying one remote database to another remote database
And the solution is here:
mysqldump --single-transaction -u (old_database_username) -p -h (old_database_host) (database_name) | mysql -h (new_host) -u (new_user) -p -D (new_database)
But even if I run this, I'm still running into an issue with credentials. The execution wants both passwords at the same time, for old DB and new DB so it keeps failing.
I tried to inline the -p but it still asks for password. What am I missing?
Okay, that was a silly mistake. The reason I was having issues is that after option such as -u or -h, there is a space while in the option for password, there is no space. I.E.
mysqldump --single-transaction -u old_database_username -pPasswordOld -h old_database_host database_name | mysql -h new_host -u new_user -pPasswordNew -D new_database
Once corrected, everything was done.
I'm trying to pull a heroku database to my local Windows computer by using heroku bash command
heroku pg:pull HEROKU_POSTGRESQL_COLOR mydatabase --app appname,
when I running above command I get the following error:
'env' is not recognized as an internal or external command, operable program or batch file.!
But local database 'mydatabase' is created, but without any tables.
My heroku app's database has a table in it, but it is not getting pulled to my local database.
Help me to solve it.
a couple of things:
1.When there is an error such as "'env' is not recognized as an internal or external command, operable program or batch file" it means that the system is trying to execute a command named env. This has nothing to do at all with setting up your environment variables.
Env is not a command in windows, but in unix. I understand that you have a windows machine though. What you can do is run "git bash". (You could get it by itself but it comes with Heroku's CLI).
This gives you a unix-like environment where the "env" command is supported, and then you can run the actual heroku pg:pull command.
2.If that still doesn't work, there is a workaround which works,without installing anything extra. Actually this is based on a ticket which I submitted to Heroku so I'm just going to quote their response:
"The pg:push command is just a wrapper around pg_dump and pg_restore commands. Due to the bug you encountered, it sounds like we should go ahead and do things manually. Run these using cmd.exe (The Command Prompt application you first reported the bug). First grab the connection string from your heroku application config vars.
heroku config:get DATABASE_URL
Then you want to pick out the username / hostname / databasename parts from the connection string, ie: postgres:// username : password # hostname : port / databasename. Use those variables in the following command and paste in the password when prompted for one. This will dump the contents of your heroku database for a local file.
pg_dump --verbose -F c -Z 0 -U username -h hostname -p port databasename > heroku.dump
Next you will load this file into your local database. One thing that the CLI does before running this command is to check and make sure the target database is empty, because running this against a database with real data is something you want to avoid so be careful with pg_restore. When running this manually you run the risk of mangling your data without the CLI check, so you may want to manually verify that the target database is empty first.
pg_restore --verbose --no-acl --no-owner -h localhost -p 5432 -d mydb2 < heroku.dump
I am sorry this is not a better experience, I hope this will help you make progress. We are in the process of rewriting our pg commands so that they work better on all platforms including windows, but there is no solid timeline for when this will be completed."
For taking backup like dump file in heroku firstly you need the backups addon, installing..
$heroku addons:add pgbackups
Then running below command will give you dump file in the name of latest
$ heroku pgbackups:capture
$ curl -o latest.dump `heroku pgbackups:url`
or
wget "`heroku pgbackups:url --app app-name`" -O backup.dump
Edited:(After chatting with user,)
Problem: 'env' is not recognized as an internal or external command, operable program or batch file.!
I suspected that one of the PATH variable to particular program is messed up. You can double click and check that in WINDOWS\system32 folder.
Ok so How to edit it:
My Computer > Advanced > Environment Variables
Then choose PATH and click edit button
Is there a Heroku way to backup / restore a standalone postgres database at https://postgres.heroku.com/ apart from using pg_dump / pg_restore ? The database is not attached to any app.
I can create a dump using
pg_dump --verbose -F c -b -h hostname -p port -U username -f "backup.dump" database_name
and restore it using
pg_restore --verbose --clean --no-acl --no-owner -h hostname -p port -U username -d database_name "backup.dump"
Looks when you create a new database, heroku automatically creates an app for you. Check it with heroku list So you can restore using the empty app.
heroku pgbackups:restore DATABASE 'dumped-db-secure-url' --app heroku-postgres-xxx