I'm trying to send logs from an Oracle RDS hosted in Amazon to an S3 bucket. I'd like to send logs to the S3 bucket daily.
What would be a recommended course of action to achieve this? I'm not concerned if the data is compressed or in it's original format.
I'm also relatively new to AWS so I'm not fully aware of all the features that are available which could make this possible if there are any.
There are 2 ways you can do that:
Download the log file using instructions here: http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.html#USER_LogAccess.Procedural.Downloading and then upload it to S3
Automate the process of downloading the log file using CLI (check the above link for CLI commands) and upload to S3.
Related
Problem
Is there a way to achieve transactionality between S3 and another database like ElasticSearch?
What I'm trying to do is to upload an object to S3 and save his identifier to ElasticSearch in an atomic way.
For the backend where logic exists, we are using Java with Springboot.
From AWS docs
I saw that this is a common pattern recommended by AWS, but they mention that you need to handle on our own the failures:
"You can also store the item as an object in Amazon Simple Storage Service (Amazon S3) and store the Amazon S3 object identifier in your DynamoDB item."
"DynamoDB doesn't support transactions that cross Amazon S3 and DynamoDB. Therefore, your application must deal with any failures, which could include cleaning up orphaned Amazon S3 objects."
Ref: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-use-s3-too.html
I have a high voluminous data in my oracle database. I want to migrate it on the AWS S3 bucket. I cannot find a good documentation for this. Please share if someone has already done it.
Thanks
You can use AWS Data Pipeline
[Copied from above link]
With AWS Data Pipeline, you can regularly access your data where it’s stored, transform and process it at scale, and efficiently transfer the results to AWS services such as Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon EMR.
Also found some code on GitHub to backup Oracle data to S3 link
I've launched an application in AWS -> Beanstalk using pre-installed server template.
In the process of Beanstalk installation I see it is creating S3 bucket. I'm pretty sure that I didn't select any option to use S3 bucket. If S3 bucket is needed for the Beanstalk application, can you tell me how it works together and what is the purpose? Can I prevent using S3 with Beanstalk?
This S3 bucket is indeed automatically created by Elastic Beanstalk for your new application.
It is used to store some environment files, and more important, zipped builds of your app (each one being a different version). The Beanstalk deployment script simply downloads the .zip from the bucket to the EBS volume.
It looks like there is no option on AWS to change this.
By the way, why don't you want to use S3?
I am looking to have multiple Amazon EC2 instances use the same data store. Amazon does not condone mounting an S3 Bucket as a file system, so I am trying to avoid this solution. Is there a way to synchronize an EBS volume with S3 or would it be best to use rsync and cron?
Do you really have to have the files locally available from within EBS? What if instead you served them to yourself via CloudFront, and restricted the permissions so that only your instances (or only your Security Group) could see the files?
Come Fall 2015 you'll be able to use Elastic File Storage (EFS) for this. But until then, I would suppose the next best thing is to use the AWS command-line to sync down from S3 to your volume:
aws s3 sync s3://my-bucket/my/folder /mnt/my-ebs/
After the initial run, that sync command is surprisingly fast. So from there you could just cron it to run hourly or so?
I am in process of hosting a dynamic website on Amazon EC2. I have created the environment and deployed war on ElasticStalkBean. I can connect to mysql database too. But I am not sure how my web application will read/write to the disk and at which path?
As per my understanding, Amazon provides 3 options for file storage
S3
EBS (Persistant)
instance storage
I could upload files on s3 creaing bucket but how can my web application read or write to S3 bucket path on differnt server?
I am not sure how should i upload files or write file to EBS. Connecting to EC2, I cannot cd /dev/sd* directory for my EBS attached to my environment instance. How can I configure my web app to use this as directory for images etc
Instance storage is lost if I stop or recreate env. and is non persistant. So not interested to store files here.
Can you help me on this?
Where to upload file that are read by application?
Where can my application write files?
Your question: "how can my web application read or write to S3 bucket path on different server?
I'm a newbie user of AWS too, so can only offer limited help, but this is what I understand:
The webapp running in the EC2 instance can access the S3 storage using with the REST or SOAP APIs. Here's the link to the reference guide for using the REST GET function to get a file from S3:
GET object documentation
I guess the idea is that the S3 storage bucket that Amazon create for your EBS "environments" provides permanent storage for your application and data files (images etc.). When a EC2 instance is created or rebooted, it should get any additional application files from an S3 bucket and 'cache' them on the file system ("volume") attached to the EC2 "instance".