I am using AWS-CLI in windows cmd and run AWS s3 sync command but it does not work with --recursive, it shows unknown options: --recursive
aws s3 sync --recursive localpath s3://bucket-name
python --version python 3.6.5
aws --version aws-cli/1.15.38 Python/2.7.9 Windows/2012Server botocore/1.10.38
Please help
The aws s3 sync command is already recursive, so there is no need for a recursive option, and there isn't one:
Syncs directories and S3 prefixes. Recursively copies new and updated
files from the source directory to the destination. Only creates
folders in the destination if they contain one or more files.
In addition the sync command only copies things that don't already exist on the destination. If you point to a folder it will recursively sync everything inside that doesn't already exist on your target destination. This is different then the aws s3 cp command. The cp command copies whatever you tell it to, regardless of it it already exists on the target. The cp command takes a --recursive option for recursively copying folders.
Related
I am trying to copy or sync newly added or modified files only from local folder to GCS bucket using rsync, It looks like rsync is copying all the files again to gcs buckets instead of copying updated or newly added files.
The command I am using right now :
gsutil -m rsync -r -d ./data/ gs://bucket/data/
Any help would be appreciated.
There are files from an AWS s3 bucket that I would like to download, they all have the same name but are in different subfolders. There are no credentials required to download and connect to this bucket. I would like to download all the files called "B01.tif" in s3://sentinel-cogs/sentinel-s2-l2a-cogs/7/V/EG/, and save them with the name of the subfolder they are in (for example: S2A_7VEG_20170205_0_L2AB01.tif).
Path example:
s3://sentinel-cogs/sentinel-s2-l2a-cogs/7/V/EG/2017/2/S2A_7VEG_20170205_0_L2A/B01.tif
I was thinking of using a bash script that prints the output of ls to download the file with cp, and save it on my pc with a name generated from the path.
Command to use ls:
aws s3 ls s3://sentinel-cogs/sentinel-s2-l2a-cogs/7/V/EG/2017/2/ --no-sign-request
Command to download a single file:
aws s3 cp s3://sentinel-cogs/sentinel-s2-l2a-cogs/7/V/EG/2017/2/S2A_7VEG_20170205_0_L2A/B01.tif --no-sign-request B01.tif
Attempt to download multiple files:
VAR1=B01.tif
for a in s3://sentinel-cogs/sentinel-s2-l2a-cogs/7/V/EG/:
for b in s3://sentinel-cogs/sentinel-s2-l2a-cogs/7/V/EG/2017/:
for c in s3://sentinel-cogs/sentinel-s2-l2a-cogs/7/V/EG/2017/2/:
NAME=$(aws s3 ls s3://sentinel-cogs/sentinel-s2-l2a-cogs/7/V/EG/$a$b$c | head -1)
aws s3 cp s3://sentinel-cogs/sentinel-s2-l2a-cogs/7/V/EG/$NAME/B01.tif --no-sign-request $NAME$VAR1
done
done
done
I don't know if there is a simple way to go automatically through every subfolder and save the files directly. I know my ls command is broken, because if there are multiple subfolders it will only take the first one as a variable.
It's easier to do this in a programming language rather than as a Shell script.
Here's a Python script that will do it for you:
import boto3
BUCKET = 'sentinel-cogs'
PREFIX = 'sentinel-s2-l2a-cogs/7/V/EG/'
FILE='B01.tif'
s3_resource = boto3.resource('s3')
for object in s3_resource.Bucket(BUCKET).objects.filter(Prefix=PREFIX):
if object.key.endswith(FILE):
target = object.key[len(PREFIX):].replace('/', '_')
object.Object().download_file(target)
I am uploading my photos from a Windows machine to an S3 bucket and each directory has a "Thumbs.db" file which I don't need to copy to the S3 bucket.
I have tried several variations of the --exclude switch to no avail. The documentation in both the help and online mentions using --exclude but there doesn't seem to be any exceptions and the Thumbs.db files keep getting uploaded. Any clues as to what I'm doing wrong?
aws s3 sync "c:/users/myaccount/pictures" "s3://bucketname/pictures" --exclude "Thumbs.db*"
But in each directory the "Thumbs.db" file keeps getting uploaded to the S3 bucket.
--exclude "Thumbs.db*" will exclude only root level Thumbs.db.* files. If they are in sub-folders, the sub-folders' names will be considered as part of the file key/name (e.g. ./a/b/Thumbs.db.info which is not coveted by "Thumbs.db*".
Thus you should try:
--exclude "*Thumbs.db*"
I have module where I need transfer all zip files from s3 bucket to my network computers by by just connecting each ip address \xx.xx.xx.xxx. right now im using laravel.
exec('aws s3 cp s3://compexp/"11-10-2019"/"01150exp.zip"');
I have bucket name: compexp inside of bucket, there are created folder name: example 11-10-2019 inside of of dated folder there are zip files for the reference see the imported image.
zip files
currently this is my reference, but i can't see how can i transfer the files from my network computers.
https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html
You can use flag --recursive together with --exclude "*" --include "*.zip" to copy only *.zip in folder "11-10-2019".
Regarding the network computers, Do you mean your internal network computers?
You can't pass another server as the destination to copy the contents. You need to copy the files locally first and the transfer to another server. A good way to sync a directory from S3 bucket contents is to use the sync command as mentioned here
aws s3 sync yourLocalDir s3://mybucket
Once you have all the contents synced up in your current directory you can just copy them to a different computer using scp command as mentioned here
scp -r yourLocalDir anotherHost:/directory
-r option of scp is to make sure you copy all the subdirectories recursively
I have a shell script that is running aws s3 cp s3://s3file /home/usr/localfile. The file already exists in that directory, so the cp command is essentially getting the latest copy from S3 to get the latest version.
However, I noticed today that the file was not the latest version; it didn't match the file on S3. Looking at the shell script's stdout from the last two runs, it looks like the command ran - the output is: download: s3://s3file to usr/localfile. But when I compared the copies, they didn't match. The changed timestamp on the file when I view it on the local machine via WinSCP (a file transfer client) didn't change either
I manually ran the command in a shell just now and it copied the file from S3 to the local machine and successfully got the latest copy.
Do I need to add a specific option for this, or is it typical behavior for files to not override a file after aws s3 cp?