Goal:
Implement reading Minio objects through signed URLs (using github.com/minio/minio-go/v6)
Attempt:
I followed the example from github:
https://github.com/minio/minio-go/blob/master/examples/s3/presignedgetobject.go
and using PresignedGetObject(), I end up with a net/url struct. When I concatenate the url.Host and url.Path value the result is something like: localhost:9000/inputs/2ea471a5521c.pdb. Which simply links to the object in Minio UI.
I expected the client to generate a signed URL that enables downloading the object when queried with curl or else, something like this in Google Cloud Storage:
https://cloud.google.com/storage/docs/access-control/signed-urls#example
Am I missing some additional logic or have I misinterpreted what minio pre-signed URLs are? Thank you.
If you want the presigned url as string you can simply call
presignedURL.String()
https://golang.org/pkg/net/url/#URL.String
Related
I have deployed an AWS Lambda function, written in Python, and AWS API Gateway structure to cause POST requests to an API endpoint to be redirected to my function. I want to upload a PDF document to my function and have it store the document in a S3 bucket. The problem I have is that the payload of any POST request to my API is being UTF-8 encoded. I don't want that but can't figure out the magic mojo to disable encoding of the request payload.
I am testing using curl, with the following command line:
curl -XPOST https://xxxxxxxxxx.execute-api.us-west-1.amazonaws.com/test -H 'content-type: application/pdf' --data-binary #document.pdf
UPDATE: I just found the following article describing how API Gateway and Lambda support uploading binary data:
https://aws.amazon.com/blogs/compute/handling-binary-data-using-amazon-api-gateway-http-apis/
This article suggests that all of the complexities that I discussed in the initial formation of my question (still provided below) should not be necessary. All I should need to do to upload binary content to my Lambda function is insure that my request includes an appropriate Content-Type header. I was already doing that, but I massaged my Curl command a bit (modified above) to define my request in exactly the way that is done in this article. I still get UTF-8 encoded data and NOT base-64 encoded data. I tried uploading a jpeg file rather than a PDF so I was doing exactly what was done in the article. Still no love. I don't get it. This article demonstrates exactly what I'm doing. But I don't get the result it suggests I should. Ggggrrrr.
ORIGINAL POST:
I am using Terraform to define my deployment. I want to cause the PDF to not be encoded/mangled at all. This is my first time using API Gateway, and I'm obviously missing some bit of config. The one thing I'm doing specifically right now to say that I want incoming payloads to be treated as binary is via the binary_media_types argument to my API definition in Terraform:
resource aws_api_gateway_rest_api proxy {
...
binary_media_types = [
"application/pdf",
"application/octet-stream",
"*/*"
]
This sets the Binary Media Types configuration associated with the API I've defined. I've confirmed via the AWS Console that this setting is having the desired effect...I can see these types in the console. I should need just the first item in the list, but I've added the others while I try to figure out the problem here. By adding that wildcard item, I believe that it shouldn't matter what the incoming Content-Type is...all payloads should be being treated as binary.
The other bit of config that I know about that might be important is the "integration contentHandling property". Here is the key bit of AWS docs that seems to explain all this:
I think the case that applies to me here is the one I've highlighted, per what I say above. This says to me that I shouldn't need to do anything else, per the "unspecified" value in the table for "contentHandling. I've tried setting the "contentHandling" argument on the integration record of my Terraform config, like this:
resource aws_api_gateway_integration proxy {
...
passthrough_behavior = "WHEN_NO_MATCH"
content_handling = "CONVERT_TO_BINARY"
}
I first tried only specifying the content_handling value. I've also tried setting that value to "CONVERT_TO_TEXT", hoping to then get base64-encoded data. Neither of these has any effect. I've tried adding the passthrough_behavior value as shown. I've also tried replacing "WHEN_NO_MATCH" with "WHEN_NO_TEMPLATES". Nothing I do changes the behavior. I haven't been able to figure out where these settings would show up in the AWS console. If I knew they were necessary, I'd explore this further. But I don't think I need to set these.
What am I missing? How can I POST a PDF document to my AWS Lambda function through API Gateway and have the payload of the request not be converted in any way? TIA!
NOTE: I am aware of this Q/A: PDF Uploaded via AWS API Gateway getting corrupted. The answer there doesn't apply to me, as I need to avoid having to form-encode the upload. The client code that will eventually be doing the upload is set in stone and sends a POST request with a payload that is just the bytes of the PDF.
I am looking to rename (or move) my S3 objects, I have their URLs like https://s3.eu-west-2.amazonaws.com/sample-bucket/temp/sample-picture.jpg
Is there any standard way in Ruby SDK to get Aws::S3::Object by only URL or I have to parse it by regular expression for example ?
In Java SDK there is AmazonS3URI.java
I haven't found any method to just take the s3 url and get the object. I ended up with parsing it by myself.
How can I generate TempURL for object on Object Storage of SoftLayer?
I currently use Ruby API and I can access my object after authenticate.
But for external End Users, I need to generate public URL for that object which is can be accessed without authentication process.
I tried to generate Temp URL with expiration, but I cannot find key for hexdigest() of HMAC. How can I found it? or is there any other way to get same result?
I can do it using the swift client like this:
We add the temporary urls secret keys with the command:
$ swift post -m "Temp-URL-Key:mykey"
We create the temporary URL:
$ swift-temp-url GET 3600 /v1/AUTH_d684780d-aafe-4772-bcbb-0f07d5f6edf3/a-container/data.txt mykey
it returns:
v1/AUTH_d684780d-aafe-4772-bcbb-0f07d5f6edf3/a-container/data.txt?temp_url_sig=19f067d38dc532883e8f02be3b43a172c61e51d2&temp_url_expires=1445615769
Then we can access to the file:
curl 'https://dal05.objectstorage.softlayer.net/v1/AUTH_d684780d-aafe-4772-bcbb-0f07d5f6edf3/a-container/data.txt?temp_url_sig=19f067d38dc532883e8f02be3b43a172c61e51d2&temp_url_expires=1445615769'
Some reference pages:
Here how to install and configure the swift client
http://sldn.softlayer.com/es/blog/waelriac/Managing-SoftLayer-Object-Storage-Through-REST-APIs
here how to create the temp URL
http://luisbg.blogalia.com/historias/74348
I hope it helps
Answer myself.
I finally wrote a code for it, after doing some more search on it including:
read related API codes from https://github.com/softlayer/softlayer-object-storage-ruby
read http://sldn.softlayer.com/es/blog/waelriac/Managing-SoftLayer-Object-Storage-Through-REST-APIs
after that, I found some hint from middle of the second document: the response Header contains this key!
X-Account-Meta-Temp-URL-Key
So, I wrote a code(method) for it and make pull request(https://github.com/softlayer/softlayer-object-storage-ruby/pull/10), and I hope it will merged soon.
anyway, If it is not accepted (yet), you can found my code at https://github.com/c12g/softlayer-object-storage-ruby
I have a mobile app that uploads images to parse.com.
I can reference the photo with a URL that looks like:
http://files.parsetfss.com/<BigLongHexString>/tfss<BigLongHexString>-Photo
BUT, for a variety of reasons, I'd like to be able to refer to that image using a different domain name so it would look more like
http://www.mydomainname.com/<BigLongHexString>/tfss<BigLongHexString>-Photo
Can this be done? I just attempted it by creating a CNAME record to redirect *.mydomainname.com to files.parsetfss.com but the result was:
<Error>
<Code>NoSuchBucket</Code>
<Message>The specified bucket does not exist</Message>
<BucketName>www.mydomainname.com</BucketName>
<RequestId>9B3C39803AABE102</RequestId>
<HostId>
5XbotEGZP/t03kgr8FkmKvyLTHN6ZBhoRcrmXU7pBn1yz1TngkulQ/RSRuAqgBxm
</HostId>
</Error>
Has anyone accomplished this? Is it possible?
I don't believe it is possible, at least the way you are trying to do it, since the files are actually stored on Amazon S3 buckets. Your domain isn't the one setup on those buckets you will get this error.
If you are using parse hosting (https://www.parse.com/docs/hosting_guide#webapp) and your domain is set point to it you can create a route to handle the redirect from there by returning the corresponding parsetfss urls in the route.
I want to upload a file to S3 using a pre-signed url
First I create a signed url using url_for(:write)
s3object = S3.buckets['myBucket'].objects['someFolder/testFile.txt']
url = s3object.url_for(:write) #
And I get the url as expected.
https://s3.amazonaws.com/myBucket/someFolder/testFile.txt?AWSAccessKeyId=AKJFGKJASGK......
But now I want to upload something using this url
I's assume that there is a way to get an s3 object from that url on which I can use the 'write' method.
Something like obj = getObjectFromUrl(url)
Is there such a method?
If not, how should I use this url to upload something to s3?
EDIT: I probably should explain.
I create the URL for testing sake.
When I will be working on it, I will only have the URL.
Using the URL alone, I will have to upload a file.
Via the current AWS SDK documentation for Ruby, the suggested method is to simply call read on your s3 object. If you already have the s3 object, you should be able to read directly from the object itself. Also, there is a method for streaming downloads if you wish to stream a file read. AWS S3 Object Documentation
obj.write('abc')
puts obj.read
#=> abc