I've started ec2 cluster with elastic-mapreduce, then logged in via ssh hadoop#ec2... and started working with grunt by pig -x local.
All is good but to access s3 storage from here I need to specify credentials in the command, like:
grunt> ls s3n:///ABRACADABRA:CADABRAABRA#domain/path/...
It is not convenient especially because it prints results with full names including these lengthy credentials.
Can I setup them somewhere to be used automatically?
If you have a EC2 instance fired up, you don't want to be running in local mode. Simply type pig in the shell. If you have an s3 bucket tied to your account, then you can cd to your bucket and access the files in it.
Once there, you could load a file like
grunt> data = load 's3://[name_of_bucket]/prod.txt' USING PigStorage(',');
Related
How to write a binary file from AWS RDS Oracle database directory to local file system on EC2. I tried using Perl script with UTL_FILE, but it can't find read the file. Getting the permissions error.
In AWS RDS Oracle, you do not have access to the file system.
If you need access to the file system then you need to use instance EC2 and install the ORACLE RDBMS.
AWS has an option to integrate with S3: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/oracle-s3-integration.html
You could upload your files there and then download to your local machine. Here are steps to use it with Datapump: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Procedural.Importing.html#Oracle.Procedural.Importing.DataPumpS3.Step1
We have a requirement to automate the process of moving the CSV files from Amazon s3 to sftp server using shell scripts(BASH). Can we achieve this requirement using shell scripting? If yes, can someone help me in sharing the sample code?
An SFTP server simply makes local files accessible via the SFTP protocol. Therefore, all you need to do is to copy the files from Amazon S3 to the local disk.
This can be done via the AWS Command-Line Interface (CLI) aws s3 cp command or, better yet, aws s3 sync. This can be called from shell scripts.
I'm attempting to use a script which automatically creates snapshots of all EBS volumes on an AWS instance. This script is running on several other instances with no issue.
The current instance already has an AWS profile configured which is used for other purposes. My understanding is I should be able to specify the profile my script uses, but I can't get this to work.
I've added a new set of credentials to the /home/ubuntu/.aws file by adding the following under the default credentials which are already there:
[snapshot_creator]
aws_access_key_id=s;aldkas;dlkas;ldk
aws_secret_access_key=sdoij34895u98jret
In the script I have tried adding AWS_PROFILE=snapshot_creatorbut when I run it I get the error Unable to locate credentials. You can configure credentials by running "aws configure".
So, I delete my changes to /home/ubuntu/.aws and instead run aws configure --profile snapshot_creator. However after entering all information I get the error [Errno 17] File exists: '/home/ubuntu/.aws'.
So I add my changes to the .aws file again and this time in the script for every single command starting with aws ec2 I add the parameter --profile snapshot_creator, but this time when I run the script I get The config profile (snapshot_creator) could not be found.
How can I tell the script to use this profile? I don't want to change the environment variables for the instance because of the aforementioned other use of AWS CLI for other purposes.
Credentials should be stored in the file "/home/ubuntu/.aws/credentials"
I guess this error is because it couldn't create a directory. Can you delete the ".aws" file and re-run the configure command? It should create the credentials file under "/home/ubuntu/.aws/"
File exists: '/home/ubuntu/.aws'
I want to back my DynamoDB local server. I have install DynamoDB server in Linux machine. Some sites are refer to create a BASH file in Linux os and connect to S3 bucket, but in local machine we don't have S3 bucket.
So i am stuck with my work, Please help me Thanks
You need to find the database file created by DynamoDb local. From the docs:
-dbPath value — The directory where DynamoDB will write its database file. If you do not specify this option, the file will be written to
the current directory. Note that you cannot specify both -dbPath and
-inMemory at once.
The file name would be of the form youraccesskeyid_region.db. If you used the -sharedDb option, the file name would be shared-local-instance.db
By default, the file is created in the directory from which you ran dynamodb local. To restore you'll have to the copy the same file and while running dynamodb, specify the same dbPath.
I am executing jmeter on AWS EC2, result of which is returned in the form csv file.
I need to upload this csv file to AWS S3 bucket.
Since I am creating number of EC2 instances dynamically and executing jmeter on those instances, it's better to automate this process .
So for this I want to write shell script (as a user data) to execute jmeter and upload result CSV file to S3 bucket from each EC2 instance.
How i can write script for this ?
Consider using command line s3 clients.
S3 command line tools
Also go through some of these sites :
Shell Script To Transfer Files From Amazon S3 Bucket.
aws command line tools
python script to upload file to s3
You can use this library for managing objects on AWS S3 using shell scripts.
Universal Docs Manager is Pure shell script based objects manager which currently supports Local Disk, MySQL and AWS S3