Am somewhat new to using Amazon S3 for image storage and just wondering the easiest way to import about about 5,000 (currently publicly available) Image URLs into S3 so that the end result is hosting of the images on our S3 Account (and resulting New URLs of the same images).
Am wondering if there is a need to first download and save each image on my computer, and then to import the resulting Images to S3 (which would of course result in new URLs for those images)? Or is there an easier to accomplish this.
Thanks for any suggestions.
James
The AWS CLI for S3 doesn’t support copying a file from a remote URL to S3, but you can copy from a local filestream to S3.
Happily, cURL outputs the contents of URLs as streams by default, so the following command will stream a remote URL to a location in S3 via the local machine:
# The "-" means "stream from stdin".
curl https://example.com/some/file.ext | aws s3 cp - s3://bucket/path/file.ext
Caveat: while this doesn’t require a local temporary file, it does ship all of the file bytes through the machine where the command is running as a relay to S3.
Add whatever other options for the S3 cp command that you need, such as --acl.
Threaded multipart transfers with chunk management will probably be more performant but this is a quick solution in a pinch if you have access to the CLI.
S3 is a “dumb” object storage, meaning that it cannot do any data processing itself, including but not limited to downloading files. So yes, you’ll have to dowload your objects first and then upload them to S3.
Related
We have found various solutions around upload the file to google cloud bucket from local system. However I am wondering if is there a way we can upload file to bucket using the public URL or link.
https://googleapis.dev/ruby/google-cloud-storage/latest/index.html
I want to upload a file from remote url to GCS bucket via ruby code. Any suggestion here would be really appreciated.
Your code sits between the remote URL and the Google Cloud Storage (GCS) Bucket.
You've 2 alternatives:
(As you describe) Download the file behind the remote URL to a file system accessible to your code and then upload it to GCS;
Stream the file from the remote location into memory (you'll need to write this) and then (using GCS client library) stream the file into a GCS object.
You tagged question with ruby-on-rails-3
Old rails versions use uploaders like carrierwave
It's possible to use it to upload files to GCS
You can upload not only local files using this gem but also from remote URL, just use special attribute
I am using AWS S3 for storing the pdf and lambda for running my functions. Recently I came across a need in which I want to convert the pdfs that are just uploaded on S3 into different images for each page. The images will be clubbed into a brand new folder created on the S3 bucket. I have almost checked everything on the internet but none of them is capable to do so.
A tool named imageMagick could help but I don't know how to implement it on lambda.
preferred language: Node.js (Optional)
I already have PlayFramework app runing , but I am in process of migrating it to Heroku. Because on Heroku I can not use local filesystem like I did in my app. I am forced to use Amazon S3 ,but I do not know how to rewrite creating of thumbnails. For that I am using :
https://github.com/coobird/thumbnailator
Thumbnails.of(picture.getFile()).size(300,300).toFile(new File("public/images/data", thumb));
The problem is that I can not do this at heroku,because file wont be saved.
How do I create thumbnails then? If I do not want to use another service which will generate thumbnails for me and save them to s3 somehow...
Honestly if I would know how many different services would I need for simple page with java then I would stay at php forever...
In Heroku (as in many PaaS) there's no persistent filesystem. However, you do have access to temp directory.
So you could save it into a temp file
File temp = File.createTempFile("prefix", "suffix").getAbsolutePath;
Thumbnails.of(picture.getFile()).size(300,300).toFile(temp, thumb));
Then take that file and upload it to S3.
If you strictly insist on not using S3 for storing binary files, then you could base64 the file content and save it into the DB. (see some pros/cons for such approach here)
Our company is using Ruby 2.1.3 with AWS SDK V1 for uploading files on S3. I need to stream files directly from a private external bucket to one of our personal bucket (without actually downloading them locally). I can't find any good documentation on the subject.
The copy_from method provided by the SDK, I think, does not permit streaming from a private external bucket to one of our bucket.
We have tried using open-uri to stream the download and stream the upload to s3 but the file was always downloaded fully first and then uploaded (is it supposed to be like that?).
Any help is welcomed!
Thank you.
The V1 SDK doesn't allow you to transfer between buckets directly. You can do what open-uri does and download the file and then upload to the new bucket.
If you want a solution that can still work in Ruby I suggest using the AWS CLI. You can add a line like this to your code:
`aws s3 cp s3://frombucket/ s3://tobucket/`
The backticks allow you to execute system commands in your ruby script. Alternatively you could upgrade to the V2 SDK and use the object.copy_to command and copy between buckets. Hope this helps!
My web application server on AWS ec2 instance.
And using MEAN stack.
I'd like to upload image to ec2 instance.(ex - /usr/local/web/images)
I can't found that how can i get the credentials.
There are just about AWS S3.
How can i upload image file to ec2 instance?
If you do file transfer repeatedly try, unison. It is bidirectional, kind of sync. Allows options to handle conflicts.
I've found the easiest way to do this as a one-off is to upload the file to google drive, and then download the file from there. View this thread to see how simply this can be done!