I have a s3 bucket and below is the directory structure.
bucketname/uid=/year=/month=/day=/files.parquet
In some cases inside year directory I have some temporary object created by athena.Ex:
month=11_$folder$
I want remove all of these files whose key = month=11_$folder$.
Currently I am doing in a loop for all uid. Is there any faster way to do that?
Using the aws cli list-objects-v2 you can search for patterns
aws s3api list-objects-v2 \
--bucket my-bucket \
--query 'Contents[?contains(Key, `month=11_$folder$`)]'
Note this will still query all your objects and only filter what is returned back, so if you have more than 1,000 objects in your bucket, you'll need to paginate
Related
I wish to move a large set of files from an AWS S3 bucket in one AWS account (source), having systematic filenames following this pattern:
my_file_0_0_0.csv
...
my_file_0_7_200.csv
Into a S3 bucket in another AWS account (target).
These need to be moved by an ec2 instance (to overcome IAM access restrictions) to an attached EBS volume incrementally (to overcome storage limitations).
Clarification:
in the filenames, there are 3 numbers separated by underscores, like so: _a_b_c, where a is always 0, b starts at 0 and goes up to 7, and c goes from 0 to maximally 200 (not guaranteed it will always reach 200).
(I have a SSH session to the EC2 instance through Putty).
1.st iteration:
So what I am trying to do in the first iteration is to copy all files from S3,
that have a name with the following pattern: my_file_0_0_*.csv.
This can be done with the command:
aws s3 cp s3://my_source_bucket_name/my_folder/ . --recursive --exclude "*" --include "my_file_0_0_*" --profile source_user
From here, I upload it to my target bucket with the command:
aws s3 cp . s3://my_target_bucket_name/my_folder/ --recursive --profile source_user
And finally delete the files from the ec2 instance's ebs volume with
rm *.
2.nd iteration:
aws s3 cp s3://my_source_bucket_name/my_folder/ . --recursive --exclude "*" --include "my_file_0_1_*" --profile source_user
This time, I only get some of the files with pattern my_file_0_1_*, as their combined file sizes reaches 100 GiB which is the limit of my ebs volume.
Here I run into the issue that the filenames are sorted alphabetically and not numerically by the digits in there names. e.g.:
my_file_0_1_0.csv
my_file_0_1_1.csv
my_file_0_1_10.csv
my_file_0_1_100.csv
my_file_0_1_101.csv
my_file_0_1_102.csv
my_file_0_1_103.csv
my_file_0_1_104.csv
my_file_0_1_105.csv
my_file_0_1_106.csv
my_file_0_1_107.csv
my_file_0_1_108.csv
my_file_0_1_109.csv
my_file_0_1_11.csv
After moving them to the target S3 bucket and removing them from ebs,
the challenge is to move the remaining files with pattern my_file_0_1_* in a systematic way. Is there a way to achieve this, e.g. by using find, grep, awk or similar ?
And do I need to cast some filename-slices to integers first ?
You can use sort -V command to consider the proper versioning of files and then invoke copy command on each file one by one or a list of files at a time.
ls | sort -V
If you're on a GNU system, you can also use ls -v. This won't work in MacOS.
I am currently working on a script that does both the lifecycle management as well as Organizing of files.
We already have lifecycle management in place, the biggest limitation I found is, lifecycle management only changes the storage class while the files are in the same place where they where.
Example
aws s3api copy-object \
--copy-source s3://dshare/folderM/alia_b_2_3 \
--key alia_b_2_3 \
--bucket s3://dshare/folderIA/alia_b_2_3 \
--storage-class STANDARD_IA
I tried the above command, however I get regex error. Let me know if i'm doing anything wrong.
Error:
Parameter validation failed:
Invalid bucket name "s3://dshare/folderIA/alia_b_2_3": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$"
My idea basically is to copy/move the files into another folder under the same or different bucket and at the same time change the Storage Class Type for the files being copied/moved.
aws s3api copy-object expects bucket name, not S3 URI starting with s3://, so in your case you need to only supply dshare as a value
I'm trying to remove only the files which are ONLY older than 5 days according to the file name containing "DITN1_" and "DITS1_" time using a bash script within the AWS S3 Bucket but the issue is all the files i'm trying to delete looks like as follows:
DITN1_2016.12.01_373,
DITS1_2012.10.10_141,
DITN1_2016.12.01_3732,
DITS1_2012.10.10_1412
if someone can help me out with the code would be nice.
thanks in advance
You can use aws cli command for deleting stuff using the bash script as follows
aws s3 rm s3://mybucket/ --recursive --include "mybucket/DITN1*"
However it does not support timestamp
For details see aws S3 cli
Is it important to use the name of the objects instead of metadata? You could get a list of objects in the bucket using the s3api:
aws s3api list-objects --bucket example --no-paginate # this last option will avoid pagination, don't use it if you have thousands of objects
Adding
--query Contents[]
Will give you back the contents of every object, including a LastModified section, which will tell you when the object was last modified, for example "2016-12-16T13:56:23.000Z".
http://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects.html
You could change this timestamp to epoch using
date "+%s" -d "put the timestamp here"
And compare it with the current time - 5 days.
OR if you really want to delete objects based on name, you could loop over the keys like this:
for key in $(aws s3api list-objects --bucket example --no-paginate --query Contents[].Key)
And add logic to determine the date. Something like this might work, judging by your examples:
key_without_prefix=${key#*_}
key_without_suffix=${key_without_prefix%_*}
Then you have your date, which you can compare with the current time - 5 days.
I am trying to find a way to perform a simultaneously copy of a AMI to all other regions.
I have search near and far but beside seeing on a blog post that it can be done, I haven't found a way using aws cli ...
https://aws.amazon.com/blogs/aws/ec2-ami-copy-between-regions/
Currently I have written a bash script to do so, but I would like to find a better, easier way to do so
I have 8 AMI's that need to be passed to all regions.
using an array-
declare -a DEST=('us-east-1' ...2....3)
aws copy-image --source-region $SRC --region ${DESTx[#]} --source-ami-id $ami
Do you guys have any other suggestion?
Thanks.
you can make a single line bash, specially useful if in future there are new regions:
aws ec2 describe-regions
--output text |\
cut -f 3 | \
xargs -I {} aws copy-image
--source-region $SRC
--region {}
--source-ami-id $ami
basically it goes like this:
aws ec2 describe-regions --output text returns the list of all available regions for ec2, its a 3 columns table ("REGIONS", endpoint, region-name)
cut -f 3 takes the 3rd column of the previous table (read as list)
keep the current region from previous argument (xargs) into {} so you can send it to the region parameter of the copy-image command
Our log files are stored in year/month/day/hourly buckets on S3. See below for structure.
How do i pass all the logs on day=20 as input to my map reduce program?
Eg:
bucket = logs/year=2014/month=8/day=20/hour=1/log1_1.txt
bucket = logs/year=2014/month=8/day=20/hour=2/log2_1.txt
bucket = logs/year=2014/month=8/day=20/hour=2/log2_2.txt
bucket = logs/year=2014/month=8/day=20/hour=2/log2_3.txt
bucket = logs/year=2014/month=8/day=20/hour=3/log3_1.txt
bucket = logs/year=2014/month=8/day=20/hour=4/log4_1.txt
When you say "bucket" do you actually mean distinct S3 buckets or do you mean folders/directories in a bucket? Creating that many buckets will end up hitting the S3 account limit for the number of buckets you can create.
Assuming you meant folders/directories in the bucket, use s3distcp as a step in your EMR cluster to copy the logs you want to HDFS and then use the HDFS directory as the input to the MR program
s3distcp takes an src directory and a srcPattern to filter the items found in src. In your example, you could do:
./elastic-mapreduce --jobflow JobFlowID --jar \
/home/hadoop/lib/emr-s3distcp-1.0.jar \
--arg --src --arg s3://logs/ \
--arg --srcPattern --arg '.*day-20.*'
--arg --dest --arg hdfs://input/
All of the logs files that have day=20 in the path will be copied to the input directory on the HDFS of the EMR cluster with JobFlowID.