I have an Amazon S3 bucket with about 300K objects in it and need to set the Cache-control header on all of them. Unfortunately it seems like the only way to do this, besides one at a time, is by copying the objects to themselves and setting the cache control header that way:
http://docs.aws.amazon.com/cli/latest/reference/s3/cp.html
Is the documentation for the Amazon S3 CLI copy command but I have been unsuccessful setting the cache control header using it. Does anyone have an example command that would work for this. I am trying to set cache-control to max-age=1814400
Some background material:
Set cache-control for entire S3 bucket automatically (using bucket policies?)
https://forums.aws.amazon.com/thread.jspa?messageID=567440
By default, aws-cli only copies a file's current metadata, EVEN IF YOU SPECIFY NEW METADATA.
To use the metadata that is specified on the command line, you need to add the '--metadata-directive REPLACE' flag. Here are some examples.
For a single file
aws s3 cp s3://mybucket/file.txt s3://mybucket/file.txt --metadata-directive REPLACE \
--expires 2100-01-01T00:00:00Z --acl public-read --cache-control max-age=2592000,public
For an entire bucket:
aws s3 cp s3://mybucket/ s3://mybucket/ --recursive --metadata-directive REPLACE \
--expires 2100-01-01T00:00:00Z --acl public-read --cache-control max-age=2592000,public
A little gotcha I found, if you only want to apply it to a specific file type, you need to exclude all the files, then include the ones you want.
Only jpgs and pngs
aws s3 cp s3://mybucket/ s3://mybucket/ --exclude "*" --include "*.jpg" --include "*.png" \
--recursive --metadata-directive REPLACE --expires 2100-01-01T00:00:00Z --acl public-read \
--cache-control max-age=2592000,public
Here are some links to the manual if you need more info:
http://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html
http://docs.aws.amazon.com/cli/latest/reference/s3/cp.html#options
Related
I have a temp directory containing two files
#Creating a temporary directory to figure out the size
tmpSizeDir=`mktemp -d`/
trap "rm -rf $tmpSizeDir" EXIT
cp -vf "${DIRECTORY}${FILE_NAME}.csv" $tmpSizeDir
cp -vf "${DIRECTORY}${FILE_NAME}.PDF" $tmpSizeDir
I also have code to send an amazon ses email
echo "{ \"Subject\": { \"Data\": \"$subject\", \"Charset\": \"UTF-8\"}, \"Body\": { \"Text\": { \"Data\": \"$body\", #\"Charset\": \"UTF-8\" } } }" > message.json
aws ses send-email --from $MAIL_SENDER --recipient file://tmpDestinationDir --message file://message.json
How do I add a directory as an attachment to an amazon ses email?
To do it: firstly you have to change the method that you are using.
aws ses send-email is not suited for attachments. If you want to use attachments, you have to use send-raw-email.
Here is the documentation for the CLI: https://docs.aws.amazon.com/cli/latest/reference/ses/send-raw-email.html
To send a directory: firstly create a zip archive, then convert the zip archive to base64 and add it to the content of the email. Afterwards, the rest in your case stays almost the same. The one thing that you have to take into account is that in sending the raw emails you are providing by yourself all of the headers, values etc, so you will have to convert your message a bit.
I wish to move a large set of files from an AWS S3 bucket in one AWS account (source), having systematic filenames following this pattern:
my_file_0_0_0.csv
...
my_file_0_7_200.csv
Into a S3 bucket in another AWS account (target).
These need to be moved by an ec2 instance (to overcome IAM access restrictions) to an attached EBS volume incrementally (to overcome storage limitations).
Clarification:
in the filenames, there are 3 numbers separated by underscores, like so: _a_b_c, where a is always 0, b starts at 0 and goes up to 7, and c goes from 0 to maximally 200 (not guaranteed it will always reach 200).
(I have a SSH session to the EC2 instance through Putty).
1.st iteration:
So what I am trying to do in the first iteration is to copy all files from S3,
that have a name with the following pattern: my_file_0_0_*.csv.
This can be done with the command:
aws s3 cp s3://my_source_bucket_name/my_folder/ . --recursive --exclude "*" --include "my_file_0_0_*" --profile source_user
From here, I upload it to my target bucket with the command:
aws s3 cp . s3://my_target_bucket_name/my_folder/ --recursive --profile source_user
And finally delete the files from the ec2 instance's ebs volume with
rm *.
2.nd iteration:
aws s3 cp s3://my_source_bucket_name/my_folder/ . --recursive --exclude "*" --include "my_file_0_1_*" --profile source_user
This time, I only get some of the files with pattern my_file_0_1_*, as their combined file sizes reaches 100 GiB which is the limit of my ebs volume.
Here I run into the issue that the filenames are sorted alphabetically and not numerically by the digits in there names. e.g.:
my_file_0_1_0.csv
my_file_0_1_1.csv
my_file_0_1_10.csv
my_file_0_1_100.csv
my_file_0_1_101.csv
my_file_0_1_102.csv
my_file_0_1_103.csv
my_file_0_1_104.csv
my_file_0_1_105.csv
my_file_0_1_106.csv
my_file_0_1_107.csv
my_file_0_1_108.csv
my_file_0_1_109.csv
my_file_0_1_11.csv
After moving them to the target S3 bucket and removing them from ebs,
the challenge is to move the remaining files with pattern my_file_0_1_* in a systematic way. Is there a way to achieve this, e.g. by using find, grep, awk or similar ?
And do I need to cast some filename-slices to integers first ?
You can use sort -V command to consider the proper versioning of files and then invoke copy command on each file one by one or a list of files at a time.
ls | sort -V
If you're on a GNU system, you can also use ls -v. This won't work in MacOS.
I have a s3 bucket and below is the directory structure.
bucketname/uid=/year=/month=/day=/files.parquet
In some cases inside year directory I have some temporary object created by athena.Ex:
month=11_$folder$
I want remove all of these files whose key = month=11_$folder$.
Currently I am doing in a loop for all uid. Is there any faster way to do that?
Using the aws cli list-objects-v2 you can search for patterns
aws s3api list-objects-v2 \
--bucket my-bucket \
--query 'Contents[?contains(Key, `month=11_$folder$`)]'
Note this will still query all your objects and only filter what is returned back, so if you have more than 1,000 objects in your bucket, you'll need to paginate
I am currently working on a script that does both the lifecycle management as well as Organizing of files.
We already have lifecycle management in place, the biggest limitation I found is, lifecycle management only changes the storage class while the files are in the same place where they where.
Example
aws s3api copy-object \
--copy-source s3://dshare/folderM/alia_b_2_3 \
--key alia_b_2_3 \
--bucket s3://dshare/folderIA/alia_b_2_3 \
--storage-class STANDARD_IA
I tried the above command, however I get regex error. Let me know if i'm doing anything wrong.
Error:
Parameter validation failed:
Invalid bucket name "s3://dshare/folderIA/alia_b_2_3": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$"
My idea basically is to copy/move the files into another folder under the same or different bucket and at the same time change the Storage Class Type for the files being copied/moved.
aws s3api copy-object expects bucket name, not S3 URI starting with s3://, so in your case you need to only supply dshare as a value
I'm trying to remove only the files which are ONLY older than 5 days according to the file name containing "DITN1_" and "DITS1_" time using a bash script within the AWS S3 Bucket but the issue is all the files i'm trying to delete looks like as follows:
DITN1_2016.12.01_373,
DITS1_2012.10.10_141,
DITN1_2016.12.01_3732,
DITS1_2012.10.10_1412
if someone can help me out with the code would be nice.
thanks in advance
You can use aws cli command for deleting stuff using the bash script as follows
aws s3 rm s3://mybucket/ --recursive --include "mybucket/DITN1*"
However it does not support timestamp
For details see aws S3 cli
Is it important to use the name of the objects instead of metadata? You could get a list of objects in the bucket using the s3api:
aws s3api list-objects --bucket example --no-paginate # this last option will avoid pagination, don't use it if you have thousands of objects
Adding
--query Contents[]
Will give you back the contents of every object, including a LastModified section, which will tell you when the object was last modified, for example "2016-12-16T13:56:23.000Z".
http://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects.html
You could change this timestamp to epoch using
date "+%s" -d "put the timestamp here"
And compare it with the current time - 5 days.
OR if you really want to delete objects based on name, you could loop over the keys like this:
for key in $(aws s3api list-objects --bucket example --no-paginate --query Contents[].Key)
And add logic to determine the date. Something like this might work, judging by your examples:
key_without_prefix=${key#*_}
key_without_suffix=${key_without_prefix%_*}
Then you have your date, which you can compare with the current time - 5 days.