Aws::S3::Errors::InvalidArgument (): on bucket.put_object - ruby

I'm trying to upload base64 encoded image to S3 bucket. My bucket list shows correctly after signing in. I'm trying to upload using the bucket#put_object
I don't see any hint as to what the invalid argument is.
My JSON body looks like..
""product": {
"product_photo": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAT8AAADQCAYAAABxw2ZIAAABQGlDQ1BJQ0MgUHJvZmlsZQAAKJFjYGASSCwoyGFhYGDIzSspCnJ3UoiIjFJgf8LAxsDMIAqEVonJxQWOAQE+QCUMMBoVfLvGwAiiL+uCzH ...."
and if I use the Carrierwave file storage, image saves fine.
How do I upload my base64 image using just the aws-sdk?
#Aws::S3::Bucket:0x00007fd60a2a8928
THat was the new bucket
UPLOADING NOW
Completed 500 Internal Server Error in 654ms (ActiveRecord: 17.3ms)
Aws::S3::Errors::InvalidArgument ():
app/controllers/products_controller.rb:139:in `supload'
def supload(params)
s3 = Aws::S3::Client.new(access_key_id: ENV['AWS_ACCESS_KEY_ID'].strip, secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'].strip) #call strip on the keys
resp = s3.list_buckets
puts resp
bucket = resp[:buckets][0]
s3Resource = Aws::S3::Resource.new
# reference an existing bucket by name
bucket = s3Resource.bucket('htm-product-photo')
puts bucket
puts "That was the new bucket"
data = Base64.decode64(params[:product_photo].to_s)
type = "image/png" #params[:contentType].to_s
extension = ".png" #params[:extension].to_s
name = ('a'..'z').to_a.shuffle[0..7].join # + ".#{extension}"
puts "UPLOADING NOW"
obj = bucket.put_object({body: data, key: name, content_type:type,acl:"public_read"})
puts obj.errors
puts "Tried to upload"
# https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Bucket.html#put_object-instance_method
url = obj.public_url().to_s
url
end
EDIT: With wire trace enabled per #D. SM
#<Aws::S3::Bucket:0x00007ffa87738708>
THat was the new bucket
UPLOADING NOW
opening connection to htm-product-photo.s3.amazonaws.com:443...
opened
starting SSL for htm-product-photo.s3.amazonaws.com:443...
SSL established, protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-GCM-SHA256
<- "PUT /ekjlaqbo HTTP/1.1\r\nContent-Type: image/png\r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby3/3.104.3 ruby/2.6.2 x86_64-darwin18 aws-sdk-s3/1.76.0\r\nX-Amz-Acl: public_read\r\nExpect: 100-continue\r\nContent-Md5: ThpRJ9rI1gkEr5zO+D4zOA==\r\nHost: htm-product-photo.s3.amazonaws.com\r\nX-Amz-Date: 20200816T221225Z\r\nX-Amz-Content-Sha256: f39e6d48f796085a4c601e08ce91be80a06fdffd22b0b3426b9eab95098fc003\r\nAuthorization: AWS4-HMAC-SHA256 Credential=AKIAUSPGIPMTP6S6J47A/20200816/us-east-1/s3/aws4_request, SignedHeaders=content-md5;content-type;host;user-agent;x-amz-acl;x-amz-content-sha256;x-amz-date, Signature=d30f1a30c6065f8d808cc238c0650c2c11799e55c2e22003571e43c90e021936\r\nContent-Length: 87850\r\nAccept: */*\r\n\r\n"
-> "HTTP/1.1 400 Bad Request\r\n"
-> "x-amz-request-id: B4DF8652D31666ED\r\n"
-> "x-amz-id-2: 32VYR2WsieoAOP2mOb27faxyhO1koISnl8ZerKLpIRQVIJE9dtZeQ3isluBzV042F1kBV7WgIEQ=\r\n"
-> "Content-Type: application/xml\r\n"
-> "Transfer-Encoding: chunked\r\n"
-> "Date: Sun, 16 Aug 2020 22:12:26 GMT\r\n"
-> "Connection: close\r\n"
-> "Server: AmazonS3\r\n"
-> "\r\n"
-> "139\r\n"
reading 313 bytes...
-> "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InvalidArgument</Code><Message></Message><ArgumentName>x-amz-acl</ArgumentName><ArgumentValue>public_read</ArgumentValue><RequestId>B4DF8652D31666ED</RequestId><HostId>32VYR2WsieoAOP2mOb27faxyhO1koISnl8ZerKLpIRQVIJE9dtZeQ3isluBzV042F1kBV7WgIEQ=</HostId></Error>"
read 313 bytes

Enable wire trace to see what is being sent to S3 and what the responses are.

The allowed options for :acl according to documentation are private, public-read, public-read-write, authenticated-read, aws-exec-read, bucket-owner-read, bucket-owner-full-control.
So, you have to change public_read to public-read
obj = bucket.put_object({body: data, key: name, content_type:type,acl:"public-read"})

Related

Does AWS apigateway change http body? How can I stop it from doing this?

Does AWS apigateway change http body? How can I stop it from doing this?
My application:
(1) A front end "UI" that sends a "http request" using "POST method" that contains a "zip file" in "body" through "form-data".
(2) AWS "apigateway" receives this request and forward it to "Lambda Proxy"
(3) AWS "Lambda" implemented by python coding receives this request and decompresses this zip file to a temporary folder.
The problem I'm facing:
(1) and (2) works fine, but in (3) the pythong program at lambda failed to decompress the file.
My finding:
It seems that when sending from the "UI" the body contains the binary data of the zip file
like below:
"PK\x03\x04\n\x00\x00\x00\x00\x00\xd6;TO\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00x2.txtPK\x03\x04\n\x00\x00\x00\x00\x00\xd6;TO\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00x1.txtPK\x01\x02\x14\x00\n\x00\x00\x00\x00\x00\xd6;TO\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00x2.txtPK\x01\x02\x14\x00\n\x00\x00\x00\x00\x00\xd6;TO\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00$\x00\x00\x00x1.txtPK\x05\x06\x00\x00\x00\x00\x02\x00\x02\x00h\x00\x00\x00H\x00\x00\x00\x00\x00"
But at (3) the python code at lambda, if we just simply returns the response like below:
response = {
"statusCode": 200,
"headers": {
"lambda-response": str(event["body"])
},
"body": "",
"isBase64Encoded": False
}
return response
will find that the binary data in the body,
seems like apigateway has changed the content
from:
"PK\x03\x04\n\x00\x00\x00\x00\x00\xd6;TO\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00x2.txtPK\x03\x04\n\x00\x00\x00\x00\x00\xd6;TO\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00x1.txtPK\x01\x02\x14\x00\n\x00\x00\x00\x00\x00\xd6;TO\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00x2.txtPK\x01\x02\x14\x00\n\x00\x00\x00\x00\x00\xd6;TO\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00$\x00\x00\x00x1.txtPK\x05\x06\x00\x00\x00\x00\x02\x00\x02\x00h\x00\x00\x00H\x00\x00\x00\x00\x00"
into:
"PK\u0003\u0004\n\u0000\u0000\u0000\u0000\u0000\ufffd;TO\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0006\u0000\u0000\u0000x2.txtPK\u0003\u0004\n\u0000\u0000\u0000\u0000\u0000\ufffd;TO\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0006\u0000\u0000\u0000x1.txtPK\u0001\u0002\u0014\u0000\n\u0000\u0000\u0000\u0000\u0000\ufffd;TO\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0006\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000
\u0000\u0000\u0000\u0000\u0000\u0000\u0000x2.txtPK\u0001\u0002\u0014\u0000\n\u0000\u0000\u0000\u0000\u0000\ufffd;TO\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0006\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000
\u0000\u0000\u0000$\u0000\u0000\u0000x1.txtPK\u0005\u0006\u0000\u0000\u0000\u0000\u0002\u0000\u0002\u0000h\u0000\u0000\u0000H\u0000\u0000\u0000\u0000\u0000\r\n"
Which is weird, what can I do to stop this?
(2019/12/17 update) below the lambda code I'm using.
import json # to decode json
import os # file IO
import shutil # file IO (use this to recursively force remove a directory)
print('Loading function')
def decompress_zip_file(src_file_path, dest_dir_path):
'''
Decompress a zip file into a directory.
Args:
src_file_path (Srting): source zip file's path.
dest_dir_path (Srting): the destination of the output directory.
Returns:
isSuccess (bool): the operation is successful or not.
'''
error_msg = "Nothing."
try:
if(os.path.isdir(dest_dir_path)):
shutil.rmtree(dest_dir_path)
with zipfile.ZipFile(src_file_path, 'r') as zip_ref:
zip_ref.extractall(dest_dir_path)
except Exception as ep:
error_msg = "Error in decompress_zip_file(), ep={}:{}".format(type(ep).__name__, str(ep))
print(error_msg)
return (False, error_msg)
return (True, error_msg)
def decompress_zip_file_from_content_in_binary(src_file_in_binary, dest_dir_path):
'''
Decompress a zip file content into a directory.
Args:
src_file_in_binary (byte array): source zip file's content in binary format.
dest_dir_path (Srting): the destination of the output directory.
Returns:
isSuccess (bool): the operation is successful or not.
'''
# write the obtained binary data into a tmp zip file
tmp_file_path = "/tmp/tmp.zip"
if(os.path.isfile(tmp_file_path)):
os.remove(tmp_file_path)
output_file = open(tmp_file_path, 'wb')
output_file.write(src_file_in_binary)
output_file.close()
(isSuccess, error_msg) = decompress_zip_file(tmp_file_path, dest_dir_path)
return (isSuccess, error_msg)
def convert_from_http_body_encoding_to_local_binary(extracted_file_from_http_body_str):
'''
Extract the file (in binary string format) from event['body'] encoding to local binary encoding.
Args:
extracted_file_from_http_body_str (string): the event['body'] file (in binary string format),.
Returns:
zipfile_binary1 (binary array): the conversion result.
'''
zipfile_binary1 = bytes(extracted_file_from_http_body_str, encoding = "ascii") # convert into a zipfile in binary format
return zipfile_binary1
def extract_zipfile_binary_from_body(body_str):
'''
Extract the zipfile (in binary format) from event['body'] string.
Args:
body_str (string): the event['body'] string.
Returns:
(binary array): the conversion result.
'''
retValue = ""
tmpArray = body_str.split("application/zip") # split the content based on MIME part field data; cut the head
if(len(tmpArray) > 1):
retValue += "entered-Lv1."
tmpArray = tmpArray[1].split("PK") # split the content based on zip file header.
if(len(tmpArray) > 1):
retValue += "entered-Lv2."
zipfile_str = "PK" + 'PK'.join(tmpArray[1:]) # add back the zip file header
tmpArray = zipfile_str.split("------WebKitFormBoundary") # split the content based on MIME part field data; cut the tail
if(len(tmpArray) > 1):
zipfile_str = tmpArray[0]
zipfile_binary = convert_from_http_body_encoding_to_local_binary(zipfile_str)
retValue = zipfile_binary
return retValue
def handler(event, context):
'''Provide an event that contains the following keys:
- operation: one of the operations in the operations dict below
- payload: a parameter to pass to the operation being performed
'''
# set the mapping table for "operation" x "return value"
operations = {
'unzip': lambda x: decompress_zip_file_from_content_in_binary(**x), # unzip an uploaded file
'ping': lambda x: 'pong' # respond to ping req.
}
# because we use "Lambda Proxe", means we have api-gateway forward the whole packet without resolving it for lambda.
event_headers = event["headers"]
operation = event_headers['operation']
event_body = event["body"]
if(operation == 'unzip'):
src_file_in_binary = extract_zipfile_binary_from_body(event_body)
payload_json = {}
payload_json['src_file_in_binary'] = src_file_in_binary
payload_json['dest_dir_path'] = "/tmp/tmp_zipfile_output"
event_headers["payload"] = payload_json
if operation in operations:
responseBody = operations[operation](event_headers.get('payload'))
response = {
"statusCode": 200,
"headers": {
"lambda-response": str(responseBody) # the api-gateway will forward the header to the front end.
},
"body": "",
"isBase64Encoded": False
}
return response
else:
raise ValueError('Unrecognized operation "{}"'.format(operation))
Below is a response from AWS support. LGTM. Leave it here so that people can see the solution to this issue in the future.
=====================Below is the response from AWS support ==================
Hi,
Thank you for contacting AWS Premium Support. I am Jyoti, and I will assist you with this case today.
From the case correspondence, I understand that you are concerned that API Gateway modifies
the binary data payload before proxying to your Lambda function. Please correct me if my understanding is wrong.
Expected Behaviour:
API gateway does modify the binary data payload into UTF-8 encoded JSON strings if
the API is configured at its default settings. Hence this is an expected behaviour.
Kindly note, as per [1], we must configure the API to support binary payloads for
our API in API Gateway. API Gateway can not send binary as is, since it has to send
a JSON body to the lambda proxy. Hence, it encodes the data/payload in UTF-8 by default.
Solution:
In order to overcome the aforementioned challenge, we need to add the desired
binary media types (application/zip in this case) to the binaryMediaTypes list
on the RestApi resource's settings page. For further information on how to achieve
this, please refer here --> [2]. If this property is not defined, the payloads
are handled as UTF-8 encoded JSON strings as mentioned in [1].
This is why the file in your request looks UTF-8 encoded. After configuring the API,
the event received by the Lambda would be a Base64-encoded string.
If you want to conduct operations on this object (the encoded request body or 'event["body"]'),
then you may decode the base64-encoded string to its orginal binary form by following
the below lines (in case of python runtime) :
import base64
coded_string = str(event["body"])
base64.b64decode(coded_string)
Troubleshooting:
I tried to replicate your setup in my environment. Instead of the frontend 'UI' of the application,
I used Postman as a client, while the rest of the setup (API Gateway and Lambda) are similar.
I am making a POST request to my API from Postman, with the request headers 'Content-Type' and 'Accept',
both set to the value 'application/zip', which is the binary media type that is being sent and
also being expected in the response. My API has been configured to support binary media types being
passed in the request body. I have added 'application/zip' in the binaryMediaTypes list for the API.
Finally, in the Lambda function I am decoding the base64-encoded request body (i.e. event["body"])
to its original binary form by using the base64 library (in python).
If you still want to confirm the consistency of your request's form-data out by returning the binary
data in your response, you can refer to the following snippet:
response {
'isBase64Encoded': True, #Ensure the body is base encoded
'statusCode': 200,
'headers': { "Content-Type": "applicaiton/zip" }, #Define the Content-Type
'body': event["body"] #Response Body returns the Base64-encoded value
}
We set the isBase64Encoded parameter to True and API Gateway automatically decodes the
response body depending on the Content-Type (i.e. the binary data/media type) that the
client (in my case Postman), is set to receive (i.e. application/zip). Kindly note, the 'Accept'
header that I had sent in my header, is to validate that the response body contains the binary
data type, the request was made for.
The above response body was the same as the request body binary data that was first sent
through the API, in my setup.
Hope I have addressed your concerns. However, if you still need help with the implementation,
please contact us again and I will be happy to assist you.
References:
=-=-=-=--=-=-=-=-=-=
[1] Support Binary Payloads in API Gateway: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-payload-encodings.html
[2] Enable Binary Support Using the API Gateway Console: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-payload-encodings-configure-with-console.html
Best regards,
Jyoti Prakash P.
Amazon Web Services
2019/12/20 update
I realize that my content type is actually multipart instead of application/zip so I modified again the setting then it worked.
Below is the help from AWS support. Many thanks to their help.
Hi,
Thanks a lot for elaborating your application flow and the logs. I understand now that your HTTP Request header 'content-type' is set to 'multipart/form-data'. I agree that for a web form to upload a file it is quite common to set content type as form-data and AWS API Gateway does support it. You would like to know if you could prevent UTF-8 encoding without changing the front end code. Please correct me if my understanding is wrong.
For the ease of discussion, I would like to separate the approach of troubleshooting for the HTTP request and response.
For the request to the API:
Please add 'multipart/form-data' as one of the values in the binaryMediaType list in your "API settings page in the API Gateway console. You would not have to alter your code or HTTP request or any of it's headers. Kindly note to handle binary media/data in API Gateway, the HTTP Request Content-Type header must match the values in binaryMediaType list.
In your use case, if you want to send the binary media back in a response for your request, the HTTP Request 'Content-Type' and 'Accept' headers, the binaryMediaType value of the API and the HTTP Response 'Content-Type' must all be set to 'multipart/form-data'. I tried the above and it worked for me with Postman Client. The 'boundary' directive is set up by Postman automatically if the HTTP Request 'Content-Type' is set to 'multipart/form-data'. Hence, you would have to only add 'multipart/form-data' in the 'binaryMediaType' list. Please have a look at my HTTP request, below:
POST /stg-with-logs HTTP/1.1
Host: <some-api-id>.execute-api.us-east-1.amazonaws.com
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
Accept: multipart/form-data
Cache-Control: no-cache
Postman-Token: 123b64f9-5669-f794-b9df-34a7561e9708
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="File"; filename="archive.zip"
Content-Type: application/zip
------WebKitFormBoundary7MA4YWxkTrZu0gW--
For the response from the API:
I noticed while going through your API Gateway Logs, the header 'isBase64Encoded' was not set. Kindly set that to true. API Gateway automatically decodes any base64-encoded string in the body of your HTTP response, if 'isBase64Encoded' is set to true. Please have a look at the HTTP Response from my lambda below:
(a6729f56-b245-45a4-9ac4-7e00b23c8957) Endpoint response body before transformations:
{
"isBase64Encoded": true,
"statusCode": 200,
"headers": {
"Content-Type": "multipart/form-data",
"Accpet": "multipart/form-data"
},
"body": "LS0tLS0tV2ViS2l0Rm9ybUJvdW5kYXJ5SmxkSW1aV1lHczlSTndPWQ0KQ29udGVudC1EaXNwb3NpdGlvbjogZm9ybS1kYXRhOyBuYW1lPSJGaWxlIjsgZmlsZW5hbWU9ImFyY2hpdmUuemlwIg0KQ29udGVudC1UeXBlOiBhcHBsaWNhdGlvbi96aXANCg0KUEsDBBQAAAAIAKF4kE9/Mo7/XgAAAJcAAAAaABwASGVsbG8tV29ybGQtNjY3MzMxNTI4MS50eHRVVAkAA8ZP910SUPdddXgLAAEEHZHreQTMewNxNY1BDgIxDAPvvIVPOY3SEC+9WCrfJ13EZWTNHKwKkzMmxIp5dpsnFMlqrjzBF/SKxCW2/8dl3ttGGjTqnkdMG+Wwj96jA3/YJsC2QF9iesuLUXPfv80KrpaVYeDjC1BLAQIeAxQAAAAIAKF4kE9/Mo7/XgAAAJcAAAAaABgAAAAAAAEAAACkgQAAAABIZWxsby1Xb3JsZC02NjczMzE1MjgxLnR4dFVUBQADxk/3XXV4CwABBB2R63kEzHsDcVBLBQYAAAAAAQABAGAAAACyAAAAAAANCi0tLS0tLVdlYktpdEZvcm1Cb3VuZGFyeUpsZEltWldZR3M5Uk53T1kNCkNvbnRlbnQtRGlzcG9zaXRpb246IGZvcm0tZGF0YTsgbmFtZT0iVGVzdCBEYXRhIg0KDQpUZXN0aW5nIEJvdW5kYXJ5IGluIG11bHRpcGFydC9mb3JtLWRhdGENCi0tLS0tLVdlYktpdEZvcm1Cb3VuZGFyeUpsZEltWldZR3M5Uk53T1ktLQ0K"
}
Along with this correspondence I am attaching my API Gateway Swagger file and Lambda function code for your reference. The setup worked fine for me and I was able to return the binary payload upon making an HTTP Request. If you want to test it out in your environment, please set the appropriate credentials and lambda uri in the Swagger file.
Hope this addresses your concern. However, if the issue still persists or you have any further questions, please contact us again and I will be happy to assist you.
To see the file named 'binaryPost-stg-with-logs-oas30-apigateway.yaml,python-binary-response.py' included with this correspondence, please use the case link given below the signature.
Best regards,
Jyoti Prakash P.
Amazon Web Services
Check out the AWS Support Knowledge Center, a knowledge base of articles and videos that answer customer questions about AWS services: https://aws.amazon.com/premiumsupport/knowledge-center/?icmpid=support_email_category

Receiving an image using Python 3 Sockets

I'm having a bit of a hard time receiving an image using sockets. I think the problem is related to the fact that sockets send both a header and the actual image, and that the two need different decoding.
This is the code:
import socket
mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysock.connect(('www.py4inf.com', 80))
mysock.send(
'GET http://www.py4inf.com/cover.jpg HTTP/1.0\n\n'.encode('utf-8'))
count = 0
fhand = open("stuff.jpg", "wb")
while True:
data = mysock.recv(512)
if len(data) < 1:
break
fhand.write(data)
mysock.close()
fhand.close()
Yes, there is a header. The end of it is after the first \r\n\r\n sequence. Once you see that sequence send the rest to a file. Here's a crude fix:
import socket
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as mysock:
mysock.connect(('www.py4inf.com', 80))
mysock.send(b'GET http://www.py4inf.com/cover.jpg HTTP/1.0\n\n')
header = b''
while True:
data = mysock.recv(512)
if not data:
raise RuntimeError('no header?')
header += data
# end-of-header in buffer yet?
eoh = header.find(b'\r\n\r\n')
if eoh != -1:
break
# split the header off and keep data read after it.
header,data = header[:eoh+4],header[eoh+4:]
print(header.decode())
with open("stuff.jpg", "wb") as fhand:
fhand.write(data)
while True:
data = mysock.recv(512)
if not data:
break
fhand.write(data)
Here's the header. Note that the content length is in the header, so if you were to send an HTTP request with a keepalive, you would have to read exactly that many bytes after the header. since Connection: close is specified, you only have to read until no more data is received.
HTTP/1.1 200 OK
Date: Sun, 22 May 2016 23:22:20 GMT
Server: Apache
Last-Modified: Fri, 04 Dec 2015 19:05:04 GMT
ETag: "b294001f-111a9-526172f5b7cc9"
Accept-Ranges: bytes
Content-Length: 70057
Connection: close
Content-Type: image/jpeg

Upload bytes or base64 string to s3 ruby sdk

I'm trying to upload an image to s3 using the ruby aws sdk. I'm able to upload the base64 string if I don't set the content_type. If I do set the content_type to image/png the upload is just the generic image thumbnail.
obj = #<Aws::S3::Object bucket_name="mybucket", key="test">
>> params[:file]
>> "data:image/png;base64,iVB...."
obj.put(body: params[:file], content_type: 'image/png', content_encoding: 'base64')
How can I upload a Base64 string to s3? I'm also open to uploading as bytes if that's more straight forward
I managed to get it to work with put, instead of upload_file. put makes more sense, since you are trying to upload an image encoded as a base64 data uri string.
Assuming your data uri string looks like:
string = 'data:image/jpeg;base64,<base_64_string>'
The key is to decode the <base_64_string> part and send it to body in the put method:
body = Base64.decode64(string.split(',')[1])
s3 = Aws::S3::Resource.new
s3.bucket(ENV['AWS_BUCKET_NAME']).object(key).put(body: body, acl: 'public-read', content_type: 'image/jpeg', content_encoding: 'base64')
This assumes you are using the the AWS Ruby library at https://github.com/aws/aws-sdk-ruby
I managed to solve this using the following code
file = params[:file]
s3 = Aws::S3::Resource.new
obj = s3.bucket('my_bucket').object("my_path").upload_file(file.tempfile)

Ruby How to send image over tcp (http) server to client

I am trying to create a simple http server but when i send an png file firefox tells me that there are problems with the image. (the image can not be display).
I changed the content type to octed-stream so i can download the file from the browser.
By comparing the original image and the downloaded image with the text editor I realized that the downloaded one missed some lines in the beginning.
# Provides TCPServer and TCPSocket classes
require 'socket'
server = TCPServer.new('localhost', 1234)
ary = Array.new
f = File.binread '/Users/serrulez/Documents/GIT/KOS_Simple_HTTP_Server/Shrek.png'
loop do
socket = server.accept
request = socket.gets
index1 = request.index(' ');
index2 = request.index(' ',index1+1);
index3 = request.index("\r\n");
index4 = request.length;
method = request[0,index1]
URI = request[index1+1,index2-index1-1]
version = request[index2+1,index3-index2-1]
CRLF = b2s((request[index3,2] == "\r\n"))
if (URI == "/Shrek" || URI == "/index.html")
socket.print "HTTP/1.1 200 OK\r\n" +
"Content-Type: image/png\r\n" +
"Connection: close\r\n\r\n"
ary = Array.new
f = File.binread '/Users/serrulez/Documents/GIT/KOS_Simple_HTTP_Server/Shrek.png'
#if I print 'f' here to the console i get the hole image, but downloaded from the browser
#the upper part ist cut
puts f
ary.push(f)
socket.write(ary) # <- update, forgot to copy this line
end
socket.close
end
I now get the whole image with the above code but my characters are escaped.
The original image starts with :
�PNG
IHDR�H
�_jsRGB���gAMA���a
[...]
and the image I receive with the browser starts with;
["\x89PNG\r\n\x1A\n\x00\x00\x00\rIHDR\x00\x00\x00\xF3\x00\x00\x01H\b\x02\x00\x00\x00\n\xEE_j\x0
I solved the problem by myself. The last piece to add was the encoding (see below)
if (URI == "/Shrek")
socket.print "HTTP/1.1 200 OK\r\n" +
"Content-Type: image/png; charset=utf-8\r\n" +
"Connection: close\r\n\r\n"

Is it possible to rename a file name before user right-click's + save as with Carrierwave + S3 + Heroku + Content-Disposition?

Is it possible to rename a file name before user right-click's + save as with Carrierwave + S3 + Heroku + Content-Disposition? I'm thinking of sanitizing file names (e.g. 193712391231.flv) before the file is saved on the S3 server and saving the original file name in a column in the db.
When a user decides to download the file (right-click and save as). I can't serve / send it as 193712391231.flv. Instead, I would like to send the file with its original file name.
How can this be implemented?
Using Carrierwave. I've come across this:
uploaded = Video.first.attachment
uploader.retrieve_from_store!(File.basename(Video.first.attachment.url))
uploader.cache_stored_file!
send_file uploader.file.path
This wont be served by S3, because it first caches the file in the local filesystem and then sends it to the browser. Which takes up a whole web process (Dyno in Heroku).
If anyone has any ideas, please suggest.
Akshully, you can:
# minimal example,
# here using mongoid, but it doesn't really matter
class Media
field :filename, type: String # i.e. "cute-puppy"
field :extension, type: String # i.e. "mp4"
mount_uploader :media, MediaUploader
end
class MediaUploader < CarrierWave::Uploader::Base
# Files on S3 are only accessible via signed URLS:
#fog_public = false
# Signed URLS expire after ...:
#fog_authenticated_url_expiration = 2.hours # in seconds from now, (default is 10.minutes)
# MIME-Type and filename that the user will see:
def fog_attributes
{
"Content-Disposition" => "attachment; filename*=UTF-8''#{model.filename}",
"Content-Type" => MIME::Types.type_for(model.extension).first.content_type
}
end
# ...
end
The url that model.media.url yields will then return the following headers:
Accept-Ranges:bytes
Content-Disposition:attachment; filename*=UTF-8''yourfilename.mp4
Content-Length:3926746
Content-Type:video/mpeg
Date:Thu, 28 Feb 2013 10:09:14 GMT
Last-Modified:Thu, 28 Feb 2013 09:53:50 GMT
Server:AmazonS3
...
The browser will then force a download (instead of opening in browser) and use the filename you set, regardless of the file name use used to store stuff in the bucket. The only drawback of this is that the Content-Disposition header is set when Carrierwave creates the file, so you couldn't for example use different filenames on the same file for different users.
In that case you could use RightAWS to generate the signed URL:
class Media
def to_url
s3_key = "" # the 'path' to the file in the S3 bucket
request_header = {}
response_header = {
"response-content-disposition" => "attachment; filename*=UTF-8''#{filename_with_extension}",
"response-content-type" => MIME::Types.type_for(extension).first.content_type
}
RightAws::S3Generator.new(
Settings.aws.key,
Settings.aws.secret,
:port => 80,
:protocol => 'http').
bucket(Settings.aws.bucket).
get(s3_key, 2.hours, request_header, response_header)
end
end
EDIT: It is not necessary to use RightAWS, uploader#url supports overriding response headers, the syntax is just a bit confusing (as is everything with CarrierWave, imho, but it's still awesome):
Media.last.media.url(query: {"response-content-disposition" => "attachment; filename*=UTF-8''huhuhuhuhu"})
# results in:
# => https://yourbucket.s3.amazonaws.com/media/512f292be75ab5a46f000001/yourfile.mp4?response-content-disposition=attachment%3B%20filename%2A%3DUTF-8%27%27huhuhuhuhu&AWSAccessKeyId=key&Signature=signature%3D&Expires=1362055338
If you're sending the file to the user direct from S3 you've no option. If you route the file through a dyno you can call it what you want, but you're using a dyno for the entire duration of the download.
I would store the files in a user friendly manner where possible and use folders to organise them.

Resources