Transfer large number of large files to s3 - bash

Question

I am transferring around 31 TB of data that consists of 4500 files, file sizes range from 69MB to 25GB, from a remote server to a s3 bucket. I am using s4cmd put to do this and put it in a bash script upload.sh:
#!/bin/bash
FILES="/path/to/*.fastq.gz"
for i in $FILES
do
echo "$i"
s4cmd put --sync-check -c 10 $i s3://bucket-name/directory/
done
Then I use qsub to submit the job:
qsub -cwd -e error.txt -o output.txt -l h_vmem=10G -l mem_free=8G -l m_mem_free=8G -pe smp 10 upload.sh
This is taking way too long - it took 10 hours to upload ~20 files. Can someone suggest alternatives or modifications to my command?
Thanks!

Answer 1

Your case may belong to the situation when copying the data onto physical media and shipping it by regular mail is faster and cheaper than transferring the data over the internet. AWS supports such a "protocol" and has a special name for it - AWS Snowball.
Snowball is a petabyte-scale data transport solution that uses secure
appliances to transfer large amounts of data into and out of the AWS
cloud. Using Snowball addresses common challenges with large-scale
data transfers including high network costs, long transfer times, and
security concerns. Transferring data with Snowball is simple, fast,
secure, and can be as little as one-fifth the cost of high-speed
Internet.
With Snowball, you don’t need to write any code or purchase any
hardware to transfer your data. Simply create a job in the AWS
Management Console and a Snowball appliance will be automatically
shipped to you*. Once it arrives, attach the appliance to your local
network, download and run the Snowball client to establish a
connection, and then use the client to select the file directories
that you want to transfer to the appliance. The client will then
encrypt and transfer the files to the appliance at high speed. Once
the transfer is complete and the appliance is ready to be returned,
the E Ink shipping label will automatically update and you can track
the job status via Amazon Simple Notification Service (SNS), text
messages, or directly in the Console.
* Snowball is currently available in select regions. Your location will be verified once a job has been created in the AWS Management
Console.
The capacity of their smaller device is 50TB, a good fit for your case.
There is also a similar service AWS Import/Export disk, where you ship your own hardware (hard drives), instead of their special device:
To use AWS Import/Export Disk:
Prepare a portable storage device (see the Product Details page for supported devices).
Submit a Create Job request. You’ll get a job ID with a digital signature used to authenticate your device.
Print out your pre-paid shipping label.
Securely identify and authenticate your device. For Amazon S3, place the signature file on the root directory of your device. For
Amazon EBS or Amazon Glacier, tape the signature barcode to the
exterior of the device.
Attach your pre-paid shipping label to the shipping container and ship your device along with its interface connectors, and power supply
to AWS.
When your package arrives, it will be processed and securely
transferred to an AWS data center, where your device will be attached
to an AWS Import/Export station. After the data load completes, the
device will be returned to you.