I've uncommented the backup_bucket: line in rubber.yml, and now my db gets backed up both locally and to my S3 bucket. I would like to have my db only backing up to S3. Is there a way to disable local backup, while still keeping S3 backup?
The only way I was able to do this, was add the following to the db backup crontab job (config/rubber/role/db/crontab):
&& rm -rf /mnt/db_backups/*
Related
I got a Windows Server 2012 R2 EC2 instance and fail to import txt-files from a S3 bucket.
I want to set up a regular data import from an S3 bucket to the EC2 instance using the aws-cli. To test the command, I opened the command prompt with administration rights, navigated to the directory, where I want to import the files and run the following command.
aws s3 cp s3://mybuckt/ . --recursive
Then I get an error like the following for every file in the bucket:
download failed: s3://mybuckt/filename.txt to .\filename.txt [Error 87] The parameter is incorrect
I end up with a list of empty files in my directory. The list is equal to that on the bucket but the text files are plain empty.
When I try the command without recursive, nothing happens. No error messages, no files copied.
aws s3 cp s3://mybuckt/ .
Here are my questions:
Why is the recursive option wrong when I import the files?
What can I check in the configuration of the EC2 instance, to verify that it is correctly set up for the data import?
You did not specify any files to copy. You should use:
aws s3 cp s3://mybuckt/* . --recursive
Or, you could use:
aws s3 sync s3://mybuckt/ . --recursive
The solution to my specific problem here was that I had to specify the file name on the bucket as well as on my EC2 instance.
aws s3 cp s3://mybuckt/file.txt nameOnMyEC2.txt
Heroku offers automatic and scheduled backups of your PG database.
https://devcenter.heroku.com/articles/heroku-postgres-data-safety-and-continuous-protection
GBackups will launch a dedicated dyno to take a dump of your database
and upload it to S3
Simple question: Is it possible to upload a scheduled PG backup to one's OWN S3 Bucket? Simply to have control over the backup files and to not be limited in Storage space. Researching this topic did not provide me with an answer if this is possible.
You can do it by using Heroku scheduler and a bash script.
# Set the script to fail fast if there
# is an error or a missing variable
set -eu
set -o pipefail
#!/bin/sh
# Download the latest backup from
# Heroku and gzip it
heroku pg:backups:download --output=/tmp/pg_backup.dump --app $APP_NAME
gzip /tmp/pg_backup.dump
# Encrypt the gzipped backup file
# using GPG passphrase
gpg --yes --batch --passphrase=$PG_BACKUP_PASSWORD -c /tmp/pg_backup.dump.gz
# Remove the plaintext backup file
rm /tmp/pg_backup.dump.gz
# Generate backup filename based
# on the current date
BACKUP_FILE_NAME="heroku-backup-$(date '+%Y-%m-%d_%H.%M').gpg"
# Upload the file to S3 using
# AWS CLI
aws s3 cp /tmp/pg_backup.dump.gz.gpg "s3://${S3_BUCKET_NAME}/${BACKUP_FILE_NAME}"
# Remove the encrypted backup file
rm /tmp/pg_backup.dump.gz.gpg
You can check out this tutorial for detailed step by step explanation.
One option is to create a backup (you can even create a follower database to created it from for performance reasons), then download the backup via stream to your server, and then upload it into your own S3 bucket.
If you wanted a quick Rail app to do this, you can setup https://github.com/kjohnston/pgbackups-archive. It does everything aside from creating a follower database, but if you are not too concerned with performance 24/7, then this should do fine. I don't know why Heroku doesn't offer storage to your own S3 buckets, as they store them on S3 themselves.
Here is a buildpack for doing this on a regular schedule. It hasn't been updated in a bit, but you could easily update / adapt it as needed.
I have wrote this shell script that download archives from a url list, decompresses them and finally moves them in a Cloud Storage bucket.
#!/bin/bash
# declare STRING variable
for iurl in $(cat ./html-rdfa.list); do
filename=$(basename "$iurl")
file="${filename%.*}"
if gsutil ls gs://rdfa/$file; then
echo "yes"
else
wget $iurl
gunzip $filename
gsutil cp -n $file gs://rdfa
rm $file
sleep 2
fi
done
html-rdfa.list contains the url list. The instance is created using the debian 7 image provided by gooogle.
The script run correctly for the first 5 or 6 files, but then the instance freezes and i have to delete the instance. The ram or the disk of the instance are not full when it freezes.
I think the problem is caused by the command gsutil cp, but it is strange that CPU load is practically 0 and also the RAM is free but it is impossibilo to use the instance without restarting them.
Are you writing the temporary files to the default 10GB root disk? If so, you may be running into the Persistent Disk throughput caps. To see if this is the case, create a new Persistent Disk, then mount it as a data disk and use that disk for the temporary files. Consider starting with ~200GB disk and see if that is enough throughput for your workload. Also, see the docs on Persistent Disk performance.
i am having one AWS EC2 instance. From this EC2 instance i am creating slave EC2 instances.
And while creating slave instances i am passing user data to new slave instance.In that user data i have written code for creating new directory in EC2 instance and downloading file from S3 bucket.
but problem is that, script creates new directory on EC2 instance but it Fails to download file from S3 bucket.
User Data Script :-
#! /bin/bash
cd /home
mkdir pravin
s3cmd get s3://bucket/usr.sh >> download.log
As shown above,in this code mkdir pravin create directory but s3cmd get s3://bucket/usr.sh fails to download file and download.log file also gets created but it remains empty.
How can i solve this proble, ? (AMI used for this is preconfigured with s3cmd)
Are you by chance running Ubuntu? Then Shlomo Swidler's question Python s3cmd only runs from login shell, not during startup sequence might apply exactly:
The s3cmd Python script (this one: http://s3tools.org/s3cmd ) seems to only work when run via an interactive login session, but not when run via scripts during the boot process.
Mitch Garnaat suggests that one should always beware of environmental differences inflicted by executing code within User-Data Scripts:
It's probably related to some difference in your environment when you are logged in as opposed to when the script is running as part of the startup sequence. I have run into similar problems with cron jobs.
This turned out to be the problem indeed, Shlomo Swidler summarizes the 'root cause' and a solution further down in this thread:
Mitch, your comment helped me realize what's different about the
startup sequence: the operative user is root. When I log in, I'm the
"ubuntu" user.
s3cmd looks in the current user's ~/.s3cfg - which didn't exist in
/root/.s3cfg, only in /home/ubuntu/.s3cfg.
Luckily s3cmd allow you to specify the config file's location with
--config /home/ubuntu/.s3cfg .
I am having a new sever and i want to move all my logs file from the old server to the new server on a weekly basis.
If the directory is not exist then create a directory of that week and transfer all the files of that week from the old server to the new one.
I am not able to find how to do that.
Write a cron job that triggers once every week. See this tutorial.
In your cron command, you write a copy (and optionally delete) command
scp -i private_key remote_server_address:/path/to/paste/log/dir; rm -rf /path/to/logfile/on/current/server;
done.
One thing to note, that I have used private_key to authenticate the connection. See here how to achieve password less authentication