I want to create a watcher that will automatically sync file changes from a local directory to a remote docker container. I need to find a way to transfer the files efficiently. I will also need it for a one time upload command which would transfer a complete folder from local directory to a remote docker container.
I figure one solution would be to scp to a tmp directory on the remote host, and then run docker cp via ssh to copy the files from tmp directory. Is that a good solution? Is there anything better?
By the way, if anyone knows a file sync utility for that use case, please let me know. I tried to search, but it seems like it's not the most popular development workflow?
I would tr using rsync for local to remote host syncing. From their volume mount the directory into the docker container.
I try to run the following aws cli command in console it working correctly.
I have aws access key and secret configured.
aws s3 sync "C:\uploadfolder" s3://uploadfolder
However, when i run it inside windows task scheduler in windows 10 as well as windows server 2012, I got the following error:
cannot find the file specified 0x80070002
It does not seems like it is a corrupted profile because it does not work for both windows and other command is running as expected.
Is there any step that I miss out? or any other special command needed when run aws cli in window task scheduler.
Your cli command is attempting to sync a FILE called "uploadfolder". You need to change to the directory first, then run the command. Your command should instead be:
cd C:\uploadfolder
aws s3 sync . s3://uploadfolder/
This will recursively copy all files in your local directory that are not in your s3 bucket. If you would also like the sync command to delete files that are no longer in the local directory, you also need to add the --delete flag.
aws s3 sync . s3://uploadfolder/ --delete
We have a requirement to automate the process of moving the CSV files from Amazon s3 to sftp server using shell scripts(BASH). Can we achieve this requirement using shell scripting? If yes, can someone help me in sharing the sample code?
An SFTP server simply makes local files accessible via the SFTP protocol. Therefore, all you need to do is to copy the files from Amazon S3 to the local disk.
This can be done via the AWS Command-Line Interface (CLI) aws s3 cp command or, better yet, aws s3 sync. This can be called from shell scripts.
I'm attempting to sync my bucket from my local directory using AWS Cli on Windows.
It works using the command
aws s3 sync C:\[long path name] s3://[bucket name]
I would prefer to replace the path name with something shorter or just associate it with the bucket. I've tried chdir and cd. Is there an easy way to do this?
If you wish to synchronize the current directory to an Amazon S3 bucket, use:
aws s3 sync . s3://[bucket name]
Or to sync to a directory within the S3 bucket:
aws s3 sync . s3://[bucket name]/[path]
The same syntax works on Windows and Linux.
If I want to download all the contents of a directory on S3 to my local PC, which command should I use cp or sync ?
Any help would be highly appreciated.
For example,
if I want to download all the contents of "this folder" to my desktop, would it look like this ?
aws s3 sync s3://"myBucket"/"this folder" C:\\Users\Desktop
Using aws s3 cp from the AWS Command-Line Interface (CLI) will require the --recursive parameter to copy multiple files.
aws s3 cp s3://myBucket/dir localdir --recursive
The aws s3 sync command will, by default, copy a whole directory. It will only copy new/modified files.
aws s3 sync s3://mybucket/dir localdir
Just experiment to get the result you want.
Documentation:
cp command
sync command
Just used version 2 of the AWS CLI. For the s3 option, there is also a --dryrun option now to show you what will happen:
aws s3 --dryrun cp s3://bucket/filename /path/to/dest/folder --recursive
In case you need to use another profile, especially cross account. you need to add the profile in the config file
[profile profileName]
region = us-east-1
role_arn = arn:aws:iam::XXX:role/XXXX
source_profile = default
and then if you are accessing only a single file
aws s3 cp s3://crossAccountBucket/dir localdir --profile profileName
In the case you want to download a single file, you can try the following command:
aws s3 cp s3://bucket/filename /path/to/dest/folder
You've many options to do that, but the best one is using the AWS CLI.
Here's a walk-through:
Download and install AWS CLI in your machine:
Install the AWS CLI using the MSI Installer (Windows).
Install the AWS CLI using the Bundled Installer for Linux, OS X, or Unix.
Configure AWS CLI:
Make sure you input valid access and secret keys, which you received when you created the account.
Sync the S3 bucket using:
aws s3 sync s3://yourbucket/yourfolder /local/path
In the above command, replace the following fields:
yourbucket/yourfolder >> your S3 bucket and the folder that you want to download.
/local/path >> path in your local system where you want to download all the files.
sync method first lists both source and destination paths and copies only differences (name, size etc.).
cp --recursive method lists source path and copies (overwrites) all to the destination path.
If you have possible matches in the destination path, I would suggest sync as one LIST request on the destination path will save you many unnecessary PUT requests - meaning cheaper and possibly faster.
Question: Will aws s3 sync s3://myBucket/this_folder/object_file C:\\Users\Desktop create also the "this_folder" in C:\Users\Desktop?
If not, what would be the solution to copy/sync including the folder structure of S3? I mean I have many files in different S3 bucket folders sorted by year, month, day. I would like to copy them locally with the folder structure to be kept.