How can I process every file in my S3 bucket with bash? - bash

Question

I have a bunch of large files (100MB - 1GB) in a Bucket. I would like to "map" all those files using a bash script. I am unable to download all files at once because my computer does not have enough storage.
Does anyone have an idea how I can do this? Anything smarter than the following solution?
for file in $(aws s3 ls s3://my-bucket | rev | cut -d' ' -f1 | rev) ; do
aws s3 cp s3://my-bucket/$file $file;
./script;
aws s3 cp $file s3://my-bucket/$file;
rm $file;
done
Explanation:
$(aws s3 ls s3://my-bucket | rev | cut -d' ' -f1 | rev) gets the name of the files in the bucket over which we are iterating
aws s3 cp s3://my-bucket/$file $file downloads a single file
./script runs my custom script
aws s3 cp $file s3://my-bucket/$file overwrites the old file with the new one

Answer 1

Interesting approach to reverse and reverse again - I would never have thought of that, but now one day might.
Taking the context as bash as per the title. Depending on what your script does, you could use other services like Glue for a "smarter"/faster but more complex solution.
I suppose you don't have any "folders" (prefixes) in your bucket. A more configurable and probably more reliable way to get the list to process could be to use the s3api, e.g.:
objects=$(aws s3api list-objects \
--bucket my-bucket \
--output json \
--query "Contents[].Key")
for file in $(echo $objects | jq -r '.[]'); do
aws s3 cp s3://my-bucket/$file $file;
./script;
aws s3 cp $file s3://my-bucket/$file;
rm $file;
done
This assumes you'd have jq installed.
The above doesn't need handling of "folders". You can also do some nicer things like choose a "subfolder" to process, e.g.:
--prefix path/to/process/ \
or filter on a file extension, e.g.:
--query "Contents[?contains(Key, '.mp4')].Key")
Might be nice to use prefixes to be able to handle restarts of the script or list limits being hit (e.g. default --page-size of 1000 for aws s3 ls).