I want to upload a file to S3 using a pre-signed url
First I create a signed url using url_for(:write)
s3object = S3.buckets['myBucket'].objects['someFolder/testFile.txt']
url = s3object.url_for(:write) #
And I get the url as expected.
https://s3.amazonaws.com/myBucket/someFolder/testFile.txt?AWSAccessKeyId=AKJFGKJASGK......
But now I want to upload something using this url
I's assume that there is a way to get an s3 object from that url on which I can use the 'write' method.
Something like obj = getObjectFromUrl(url)
Is there such a method?
If not, how should I use this url to upload something to s3?
EDIT: I probably should explain.
I create the URL for testing sake.
When I will be working on it, I will only have the URL.
Using the URL alone, I will have to upload a file.
Via the current AWS SDK documentation for Ruby, the suggested method is to simply call read on your s3 object. If you already have the s3 object, you should be able to read directly from the object itself. Also, there is a method for streaming downloads if you wish to stream a file read. AWS S3 Object Documentation
obj.write('abc')
puts obj.read
#=> abc
Related
Goal:
Implement reading Minio objects through signed URLs (using github.com/minio/minio-go/v6)
Attempt:
I followed the example from github:
https://github.com/minio/minio-go/blob/master/examples/s3/presignedgetobject.go
and using PresignedGetObject(), I end up with a net/url struct. When I concatenate the url.Host and url.Path value the result is something like: localhost:9000/inputs/2ea471a5521c.pdb. Which simply links to the object in Minio UI.
I expected the client to generate a signed URL that enables downloading the object when queried with curl or else, something like this in Google Cloud Storage:
https://cloud.google.com/storage/docs/access-control/signed-urls#example
Am I missing some additional logic or have I misinterpreted what minio pre-signed URLs are? Thank you.
If you want the presigned url as string you can simply call
presignedURL.String()
https://golang.org/pkg/net/url/#URL.String
I am looking to rename (or move) my S3 objects, I have their URLs like https://s3.eu-west-2.amazonaws.com/sample-bucket/temp/sample-picture.jpg
Is there any standard way in Ruby SDK to get Aws::S3::Object by only URL or I have to parse it by regular expression for example ?
In Java SDK there is AmazonS3URI.java
I haven't found any method to just take the s3 url and get the object. I ended up with parsing it by myself.
I have an Angular webapp that uses a Spring Boot REST service as its backing web service.
I am adding a "Profiles" feature for users, and as part of this I want to stand up an endpoint that allows users to upload profile images for themselves and immediately upload those files to S3 (where I will host all the images from).
Looking at several Spring Boot/file upload tutorials :
http://www.mkyong.com/spring-boot/spring-boot-file-upload-example/
I update avatar image and display it but the avatar does not change in Spring Boot , why?
Many others
It seems that the standard way of handling such file upload is exposing a controller endpoint that accepts MultipartFiles like so:
#RestController
#RequestMapping("/v1/profiles")
public class ProfileController {
#PostMapping("/photo")
public ResponseEntity uploadProfilePhoto(#RequestParam("mpf") MultipartFile mpf)
// ...
}
Looking at all this code, I can't tell if the MultipartFile instance is in-memory or if Spring sets its location somewhere (perhaps under /tmp?) on the disk.
Looking at the AWS S3 Java SDK tutorial, it seems the standard way to upload a disk-based File is like so:
File file = new File(uploadFileName);
s3client.putObject(new PutObjectRequest(bucketName, keyName, file));
So it looks like I must have a File on disk in order to upload to S3.
I'm wondering if there is a way to keep everything in memory, or whether this is a bad idea and I should stick to disks/File instances!
Is there a way to keep the entire profile image (MultipartFile) in-mempory inside the controller method?
Is there a way to feed (maybe via serialization?!) a MultipartFile instance to S3's PutObjectRequest?
Or is this all a terrible idea (if so, why?!)?
Is there a way to keep the entire profile image (MultipartFile) in-mempory inside the controller method?
No, there is NO way to keep an image File in-memory because File object in java represents a path in file system.
Is there a way to feed (maybe via serialization?!) a MultipartFile instance to S3's PutObjectRequest?
No, from S3's API documentation, there is no way for S3 to deserialize to the image file for you after/during the upload.
Or is this all a terrible idea (if so, why?!)?
It depends on your specific case but it is generally not preferred.
If - there are not many users uploading images at the same time, your memory is probably enough to handle.
Else - You can easily get out-of-memory problems.
If you insist on doing so, S3 API can upload an InputStream (If I remember correctly). You can convert your Multipart File to an InputStream.
This SO thread talks about uploading to S3 with InputStream
You can also take a look at File.createTempFile() to create a temp file.
I have been looking at the same thing. Basically you want a user to be able to be able to upload a photo album and have those photos served from S3 and probably have them secured so only that user can upload/delete/etc.
I believe the simpler answer is in spring boot to get a Pre-signed URL from S3. https://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObjectJavaSDK.html
which basically gives you a token defining the bucket, and object key ("/bobs_profile/smiling_bob.jpg") and a time limit for that image to be uploaded.
Give that to your angular app (or ionic app) to upload the image to that location.
That should do it. but someone let me know if I'm wrong.
The only issue that I see is if bob wants to upload "bobs_nude_photo.jpg" and only wants spring security logged in people to be able to see it... well I'm sure there is an S3 solution for that??
I'm making an upload form for zips in a ruby webmachine app. My idea is to have an upload through my backend where I can add some extra params and then upload it to amazons s3 service with RestClient.
I did successfully create a direct upload (web based form post) to a s3bucket, but in that way I'm unable to handle the variables which are needed in the request, the way I want.
I've tried several things but I can't figure out, how to handle the request, as soon as it gets in my backend. I've created a resource and I'm debugging directly in the process_post method.
My #request variable represents a Webmachine::Request, with a Webmachine::Adapters::Rack::RequestBody and a Rack::Request, but I can't get the file out of it to use it as input for my RestClient request.
I think; #request.body.to_s and #request.body.to_io, represent the uploaded file in some way, and I tried to use them as input for Rack::Multipart methods, but that doesn't give me the file.
I also tried to work with the rack-raw-upload gem, but I can't get the mime-type something else than "application/x-www-form-urlencoded" or multipart. I do explicitly set it to; application/octet-stream
Things like File.new(filename, 'rb') gave me `rrno::ENOENT: No such file or directory # rb_sysopen'. For filename I just used 'example.zip'.
I guess I'm missing something which has to do with the Rack::Request call(env) method.
Does somebody have an idea, on how to handle the Rack uploads? Or give me any hints for a new direction? Thanks.
I've created a gist which shows how to retrieve the multipart stream. You'll need further parsing in order to get the uploaded file.
https://gist.github.com/jewilmeer/eb40abd665b70f53e6eb60801de24342
I'm having a hell of a time working with the aws-sdk documentation, all of the links I follow seem outdated and unusable.
I'm looking for a straight forward implementation example of uploading an image file to an S3 bucket in Ruby.
let's say the image path is screenshots/image.png
and I want to upload it to the bucket my_bucket
AWS creds live in my ENV
Any advice is much appreciated.
Here is how you can upload a file from disk to the named bucket and key:
s3 = Aws::S3::Resource.new
s3.bucket('my_bucket').object('key').upload_file('screenshots/image.png')
That is the simplest method. You should replace 'key' with the key you want it stored with in Amazon S3. This will automatically upload large files for you using the multipart upload APIs and will retry failed parts.
If you prefer to upload always using PUT object, you can call #put or use an Aws::S3::Client:
# using put
s3 = Aws::S3::Resource.new
File.open('screenshots/image.png', 'rb') do |file|
s3.bucket('my_bucket').object('key').put(body:file)
end
# using a client
s3 = Aws::S3::Client.new
File.open('screenshots/image.png', 'rb') do |file|
s3.put_object(bucket:'my_bucket', key:'key', body:file)
end
Also, the API reference documentation for the v2 SDK is here: http://docs.aws.amazon.com/sdkforruby/api/index.html