I am using Sinatra with Thin.
When trying to get access the contents of a POST request with multipart data, I don't see any convenient way to access this, though it seems I can get to the raw body.
Is there something I am missing? Is there a convenient way to do this, or must I parse the data myself?
request.env['rack.request.form_hash'] gives an empty hash.
When dumping the body (request.body.read()) to stdout, it looks like this:
--KoZIhvcNAQcB
Content-Disposition: form-data; filename="1"
X-compId: 1
Content-Length: 1024
QeQL4k8RPBQVrP2u8Zf6Ie82labdA_
d7s FgvwSQvkP6e0TwaWWCfoHWbaP6
... (some lines omitted)
--KoZIhvcNAQcB--
I also see that it is not a chunked transfer encoding issue (I read that Thin has an issue with that); the Transfer-Encoding header is not set.
But as for my question, does Sinatra or Rack supply something in the API for accessing multipart details?
UPDATE: After doing some reading, I see the multipart data is supposed to be visible in the request parameters. As a result, I have logged an issue on GitHub. If I get it resolved, I will post an answer here. https://github.com/rack/rack/issues/695
Related
I have this Sinatra::Base code:
class Crush < Sinatra::Base
post '/upload' do
erb params.inspect
end
end
I am using Postman and its interface for uploading a file. So I send a POST request with form-data, where in the body of the request the name is hello and the value is a file test.txt which contains just a simple string hey there.
When I do params.inspect I get this long string
{"------WebKitFormBoundaryocOEEr26iZGSe75n\r\nContent-Disposition: form-data; name"=>"\"hello\"; filename=\"test.txt\"\r\nContent-Type: text/plain\r\n\r\nhey there\r\n------WebKitFormBoundaryocOEEr26iZGSe75n--\r\n"}
So basically a long has with a single key and a single value. Reading most Sinatra tutorials (where the file is accepted from a form), there's a nice way Sinatra handles this using params[:file], but this doesn't seem to be the case when the file is coming straight from the body of an HTTP request.
I tried a non-modular approach too withou Sinatra::Base, thinking it's some parsing middle-ware missing, but got the same result.
Is there something I'm missing here? Must I go and make my own custom parser to get the content of this long hash? Or is there an easier way?
I figured it's Postman issue. When I switch from 'x-www-form-urlencoded' to 'form-data' in Postman, in the Header section, the field: Content-Type => application/x-www-form-urlencoded is NOT removed. So for those who encounter this problem, make sure you remove it manually.
hi where is the data in post call in koa without co-body or bodyparse or why this error
Error: invalid JSON, only supports object and array
at parse (d:\Proyectos\koaJsTest\node_modules\co-body\lib\json.js:56:13)
co-body performs this regex unless the "strict" option is set to false:
/^[\x20\x09\x0a\x0d]*(\[|\{)/
Perhaps your json is making it to co-body as a URL-encoded string?
Its either going to be the format of the JSON that you are uploading if you are setting the Content-Type to application/json.
Otherwise, may be using the wrong Content-Type. For example if you were uploading files where the Content-Type should be multipart/form-data but you accidentally set the Content-Type to application/json when it should you would see this error.
This has tripped me up in the past.
We have some publicly shared S3 files that we want to make sure won't be indexed by Google. I can't seem to find any documentation on how to do this. Is there a way to set a "noindex" x-robots-tag response header on individual S3 objects?
(We're using the Ruby AWS client)
There does not appear to be a way to do this.
Only certain headers from an S3 PUT object request are documented as being returned when the object is fetched.
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html
Anything else you send appears to be simply disregarded, as long as it doesn't actually invalidate the request.
Actually, that's what I thought before researching this, and it's almost true.
The documentation here seems incomplete, and elsewhere suggests the following request headers, if sent with the upload, will appear in the download:
Cache-Control
Content-Disposition
Content-Encoding
Content-Type
x-amz-meta-*
Other headers are listed at the latter link, but some of these like Expect wouldn't make sense on a GET request, so they logically wouldn't appear.
So far, this is all consistent with my experience with S3.
If you send a random but not-invalid header with your request, it's ignored. Example:
X-Foo: bar
S3 seems to accepts this on upload, but discards it (presumably doesn't store it)... downloading the object does not return the X-Foo header.
But X-Robots-Tag appears to be an undocumented exception to this.
Uploading a file with X-Robots-Tag: noindex (for example) does indeed result in the same header and value being returned with the object when you GET it.
Unless somebody can cite the documentation that explains why this works, we're operating in distinctly undocumented territory.
But, if you're interested in going there, the simple answer appears to be, you just add this header to the HTTP PUT request you send to the REST API to upload the object.
"Not so fast," you say, "I'm using the Ruby SDK." Indeed. The AWS Ruby client seems to be too "helpful" to let you get away with this, at least, not easily. The docs there show how to add "metadata" --
:metadata (Hash) — A hash of metadata to be included with the object. These will be sent to S3 as headers prefixed with x-amz-meta. Each name, value pair must conform to US-ASCII.
Well, that's not going to work, because you'd get x-amz-meta-x-robots-tag.
How do you set other headers in the upload? Every other header you'd normally set is an element of the options hash, like :cache_control, which turns into Cache-Control: in the upload request. Unless they're blindly applying the keys from that hash to the upload transaction (which would be terrible design combined with excellent luck) then you may not have a straightforward way to get here from there. I can't be much more specific, because the only I really know about Ruby is the same thing I know about Java -- from what I've seen of it, I don't like it. :)
But X-Robots-Tag does appear to be a custom header S3 supports, to some extent, without clear documentation of that fact. It's, at least, accepted by the REST API.
Failing the above, you can manually add this header to the metadata in the S3 console after uploading the object. (Note, X-Foo: Bar doesn't work from the S3 console, either -- it's silently discarded, with no error -- but X-Robots-Tag: works fine).
You can also, of course, put a publicly-readable robots.txt file (with the appropriate directives in it) in the root of the bucket. Depending on your cobtent mix, path hierarchy, and other factors, that isn't (perhaps) as simple as selectively setting headers, but if the entire bucket is comprised of information you don't want indexed, it should easily accomplish what you want, since content should not be indexed if disallowed in robots.txt, even when a search spider follows a link to it from another site -- every domain (and subdomain)'s robots.txt file stands alone.
#Michael - sqlbot is correct. The SDKs don't support it by default and it won't show in the AWS Console, but if you set it directly with the REST API it works. For those who don't want to figure out the REST API and its authentication method, I was able to modify the node.js aws-sdk to support this feature.
Amazon stores the method params configuration and validation in a large json file: apis/s3-2006-03-01.min.json . I guess that the other SDKs may implement their validation in the same way.
You can go to the "PutObject" command, and under "input.members" you can add a new parameter "XRobotsTag". Configure it as a "header" and set the location to "X-Robots-Tag".
"XRobotsTag": {
"location": "header",
"locationName": "X-Robots-Tag"
}
Your local aws-sdk is now configured to support X-Robots-Tag on your putObject requests. In node.js this would look like this:
s3.putObject({
ACL: "public-read",
Body: "hello world",
Bucket: "my-bucket",
CacheControl: "public, max-age=31536000",
ContentType: "text/plain",
Key: "hello.txt",
XRobotsTag: "noindex, nofollow"
}, function(err, resp){});
I need to include the content-length of a image /png file posted in an upload to a webservice.
But how do I calculate the content-length to include in the header?
Thanks.
I am submitting it using rest-client.
The webservice for the upload is Postful: and the documentation has been unclear: http://www.postful.com/developer/guide#uploading_attachments
Because I am writing the payload and headers, seems like I need to input that value.
I am also looking at postalmethods which says that the content-length is the user input:
http://postalmethods.com/method/2009-02-26/UploadFile
The files themselves are .PNG. I am going to attach them to a model using Paperclip, so will have a filepath from that.
The file that I need the content-length to post is stored as an attachment using paperclip, so the specific code generating problems is:
File.size(#postalcard.postalimage.url)
Well, you know how you're reading and posting the data, presumably - so you know how much data you're sending. That's the content length. If you're just sending it directly in binary as the body of the post, it's just the length of the file. If you're base-64 encoding it, then the content length will be the ((file length + 2) / 3) * 4. If it's going in a SOAP envelope etc, you'll need to take account of that.
One way of doing this for complicated situations is to build the entire post body in memory first, set the content length, and then just copy from memory directly to the post body.
Well, you can use File.size(filepath) but it's unlikely that you'll need to - most libraries for making HTTP requests should do that automatically - which library are you using? (Or, what kind of webservice is it?)
I'm trying to implement the "blog this" function from Flickr using the BloggerAPI to my pl/sql based CMS.
When Flickr sends me the posting transaction, the HTTP transaction looks like this:
POST /pls/website/!pkg.procAPI HTTP/1.1
Host: www.mydomain.com
Accept: */*
User-Agent: Flickr
Content-Type: text/xml; charset=utf-8
Content-Length: 1220
Expect: 100-continue
<?xml version="1.0" encoding="utf-8"?>
<methodCall>
<methodName>blogger.newPost</methodName>
<params>
<param><value><string>NO_APP_KEY</string></value></param>
<param><value><string>1</string></value></param>
<param><value><string>markj</string></value></param>
<param><value><string>markj</string></value></param>
<param><value><string>This is a test post from <a href="http://www.flickr.com/r/testpost"><img alt="flickr" src="http://www.flickr.com/images/flickr_logo_blog.gif" width="41" height="18" border="0" align="absmiddle" /></a>, a fancy photo sharing thing.</string></value></param>
<param><value><boolean>1</boolean></value></param>
</params>
</methodCall>
But my server is responding with an HTTP-400 Bad Request and the error message is "Signature Mismatch or Missing '='" and my pl/sql procedure never gets a chance to process the request. I suspect that the flexible parameter passing is getting hosed when looking at the message, but I don't know how else
The process to get the available blogs seems to work ok, but the content of the request doesn't have all the html entities as part of the message:
POST /pls/website/!pkg.procAPI HTTP/1.1
Host: www.mydomain.com
Accept: */*
User-Agent: Flickr
Content-Type: text/xml; charset=utf-8
Content-Length: 304
<?xml version="1.0" encoding="utf-8"?>
<methodCall>
<methodName>blogger.getUsersBlogs</methodName>
<params>
<param><value><string>NO-APP-KEY</string></value></param>
<param><value><string>mark</string></value></param>
<param><value><string>markj</string></value></param>
</params>
</methodCall>
Is there a way to get the xml data from the body of the http request directly? or some other approach I'm over looking?
Thanks,
Mark.
I opened a service request with Oracle and they confirmed that mod_plsql does not support retrieving httpheader variables. Here is the discussion I had on the SR:
ISSUE ANALYSIS
I need to confirm it, but Mark conclusion so far looks right to me,i.e. MOD_PLSQL is always invoking and processing PLSQL procedures and there is no way to read the raw HTTP request because MOD_PLSQL parses the request and checks if there is a PLSQL procedure with this name and if there are arguments to pass. If there are no arguments to pass but we are providing something else like the XML document, MODPLSQL gets puzzled and thinks this is a new pair of parameter and value when this is not really the case.
ANSWER
No, it is not possible to do it because MOD_PLSQL just treats requests with parameters which can be parsed and translated as a PLSQL procedure call with several arguments. MOD_PLSQL has no concept of a "httprequest" variable containing the whole request in the same way java has the HTTPRequest object
ISSUE ANALYSIS
Got the confirmation unless we pass it as a parameter we can not read it from MOD_PLSQL. In other words, if we pass the XML page in the HTTP request body we need to use something like java where we can treat the request in raw mode and we can read any HTTP header and the http body without restrictions of any kind.
Can the xml be sent as a file attachment? In other words change:
Content-Type: text/xml; charset=utf-8
to
Content-Type: multipart/form-data, boundary=AaB03x
Then add this before the xml:
--AaB03x
content-disposition: form-data; name="xmlfile"; filename="myfile.xml"
Content-Type: text/xml
And this after the xml:
--AaB03x--
Now, you set up the procedure for a file upload, as explained in this article:
Files, Uploads, and Downloads with Web PLSQL
Use UTL_HTTP
You can use the UTL_HTTP package to do this.
UTL_HTTP.read_text(
r IN OUT NOCOPY resp,
data OUT NOCOPY VARCHAR2,
len IN PLS_INTEGER DEFAULT NULL);
"The UTL_HTTP package supports HTTP 1.1 chunked transfer-encoding. When the response body is returned in chunked transfer-encoding format as indicated in the response header, the package automatically decodes the chunks and returns the response body in de-chunked format."
http://download.oracle.com/docs/cd/B10500_01/appdev.920/a96612/u_http.htm
I'm encountering the same issue MojoMark was (though with a different external application). MojoMark's follow up comment is correct, the read_text information does not answer the question.
Like MojoMark, I am attempting to RECEIVE a POST from an external entity. The PLSQL I'm trying to write is to receive that POST and store it.
I was able to write the outbound POST XML with no problem, but have been unable to set up a procedure to receive it.
The issue appears to be the PLSQL procedure wants a variable name, ie:
http://server/modplsql/testload, POST "data='data'"
However, the body of the POST is just an XML data stream, no "=" assigning a value to a parameter, so the function gets the "Mismatch or Missing '='"" error.
It seems like there should be a way to tell MODPLSQL to just accept accept the post data like a "typical" function call, but I think I'm missing something obvious.
The basics function I'm trying to use to receive the post is:
procedure testload ( DATA in clob ) as
BEGIN
htp.print('Input POST stream was ' || DATA )
end testload;
The only significant difference I can see between the two requests is the "Expect: 100-continue" business. This appers to be a clause in the HTTP 1.1 specification which allows the client to split the request in two. Basically, the client sends the header to the server, and the server must make a choice. If it is "happy" with the header it will give a HTTP 100 response to indicate that the rest of the message will be accepted.
Since you're getting data from Flickr, I'm not sure what level of control you have over the http requests being made by the client. From the server side it may be necessary to make a configuration change to Apache, or it may be a bug in mod_plsql that's popping up with the "!" flexible parameter marker.
Either way, the current blogger api is not like this. You probably should move to the new GData API.