When connecting to a website using Net::HTTP you can parse the URL and output each of the URL headers by using #.each_header. I understand what the encoding and the user agent and such means, but not what the "accept"=>["*/*"] part is. Is this the accepted payload? Or is it something else?
require 'net/http'
uri = URI('http://www.bible-history.com/subcat.php?id=2')
http://www.bible-history.com/subcat.php?id=2>
http_request = Net::HTTP::Get.new(uri)
http_request.each_header { |header| puts header }
# => {"accept-encoding"=>["gzip;q=1.0,deflate;q=0.6,identity;q=0.3"], "accept"=>["*/*"], "user-agent"=>["Ruby"], "host"=>["www.bible-history.com"]}
From https://www.w3.org/Protocols/HTTP/HTRQ_Headers.html#z3
This field contains a semicolon-separated list of representation schemes ( Content-Type metainformation values) which will be accepted in the response to this request.
Basically, it specifies what kinds of content you can read back. If you write an api client, you may only be interested in application/json, for example (and you couldn't care less about text/html).
In this case, your header would look like this:
Accept: application/json
And the app will know not to send any html your way.
Using the Accept header, the client can specify MIME types they are willing to accept for the requested URL. If the requested resource is e.g. available in multiple representations (e.g an image as PNG, JPG or SVG), the user agent can specify that they want the PNG version only. It is up to the server to honor this request.
In your example, the request header specifies that you are willing to accept any content type.
The header is defined in RFC 2616.
Related
I find WebAPI separate HTTP response headers into different places, one is in Response.Headers, the other in in Response.Content.Headers. For example, etag is in Response.Headers while lastModified is in the other. What is the reason behind that?
There are a couple of answers to that question. One is because that's the way the HTTP spec defines the headers.
RFC 2616
Content header fields here
https://www.rfc-editor.org/rfc/rfc2616#section-7.1
Request header fields here https://www.rfc-editor.org/rfc/rfc2616#section-5.3
Response header fields here https://www.rfc-editor.org/rfc/rfc2616#section-6.2
The other more practical reason for separating out the content headers is that it is easier write code that processes data into HTTP payloads and sets the related headers, independent of the request/response objects.
Unfortunately, the more recent HTTPbis specification did some reorganization of where they think headers should go and now LastModified and Allow are considered response fields, not content fields.
This means that the headers as defined in System.Net.HttpHeaders will no longer match the spec, which really sucks. It also means that we are probably stuck with LastModified as a HttpContent header and Etag as a response header.
HTTPbis
Content related headers are defined here.
Request headers here.
Response headers here.
I know OData supports responding in JSON format when it's given the appropriate Accept header:
Accept: application/json
Some articles say you'll need to specify odata verbosity otherwise you'll get the default xml format, but I have not seen this to be actually true. But let me mention it anyway:
Accept: application/json;odata=verbose
But (how) can I make my request using JSON instead of a querystring?
OData doesn't provide a way to specify the query in a request body, it only supports the query in the URL. So the answer is that there's no way to do that in JSON. Note that it applies to GET requests. Modification requests (POST/PUT/...) do accept JSON as the payload (typically representing an entity for example), in which case simply specify the content type of the request in its Content-Type header.
There are java script libraries which let you build the query string using more structured code (as compared to just strings). For example the datajs http://datajs.codeplex.com/.
I'm developing a RESTful web application in Ruby with Sinatra. It should support CRUD operations, and to respond to Read requests I have the following function that formats the data according to what the request specified:
def handleResponse(data, haml_path, haml_locals)
case true
when request.accept.include?("application/json") #JSON requested
return data.to_json
when request.accept.include?("text/html") #HTML requested
return haml(haml_path.to_sym, :locals => haml_locals, :layout => !request.xhr?)
else # Unknown/unsupported type requested
return 406 # Not acceptable
end
end
Only I don't know what is best to do in the else statement. The main problem is that browsers and jQuery AJAX will accept */*, so technically a 406 error is not really the best idea. But: what do I send? I could do data.to_s which is meaningless. I could send what HAML returns, but they didn't ask for text/html and I would rather notify them of that somehow.
Secondly, supposing the 406 code is the right way to go, how do I format the response to be valid according to the W3 spec?
Unless it was a HEAD request, the response SHOULD include an entity containing a list of available entity characteristics and location(s) from which the user or user agent can choose the one most appropriate. The entity format is specified by the media type given in the Content-Type header field. Depending upon the format and the capabilities of the user agent, selection of the most appropriate choice MAY be performed automatically. However, this specification does not define any standard for such automatic selection.
It looks like you're trying to do a clearing-house method for all the data types you could return, but that can be confusing for the user of the API. Instead, they should know that a particular URL will always return the same data type.
For my in-house REST APIs, I create certain URLs that return HTML for documentation, and others that return JSON for data. If the user crosses the streams, they'll do it during their development phase and they'll get some data they didn't expect and will fix it.
If I had to use something like you're writing, and they can't handle 'application/json' and can't handle 'text/html', I'd return 'text/plain' and send data.to_s and let them sort out the mess. JSON and HTML are pretty well established standards now.
Here's the doc for Setting Sinatra response headers.
I'm crawling a site using Ruby + OpenURI + Nokogiri. Fetch a page, find all the a[href] and (if they're in the same domain and right protocol) follow them to crawl again.
Sometimes there are links to large binaries (e.g. jpeg, exe), and I don't want to crawl those.
I tried using the HTTP "Accept" header to get an error or empty response for the wrong mime types like so:
require 'open-uri'
page = open(url, 'Accept'=>'text/html,application/xhtml+xml,application/xml')
...but OpenURI still downloads binaries sent with another mime type.
Other than looking at file extensions in the url for a probable file type, how can I prevent the download (or detect a conflicting response type) for an arbitrary URL?
You could send a HEAD request first, then check the Content-type header of the response and only make the real request if it’s acceptable:
ACCEPTABLE_TYPES = %w{text/html application/xhtml+xml application/xml}
uri = URI(url)
type = Net::HTTP.start(uri.host, uri.port) do |http|
http.head(uri.path).content_type
end
if ACCEPTABLE_TYPES.include? type
# fetch the url
else
# do whatever
end
This will need an extra request for each page, but I can’t see a way of avoiding it. It also relies on the server sending the same headers for a HEAD request as it does for a GET, which I think is a reasonable assumption but something to be aware of.
I'm building some kind of proxy.
When I call some url in a rack application, I forward that request to an other url.
The request I forward is a POST with a file and some parameters.
I want to add more parameters.
But the file can be quite big. So I send it with Net::HTTP#body_stream instead of Net::HTTP#body.
I get my request as a Rack::Request object and I create my Net::HTTP object with that.
req = Net::HTTP::Post.new(request.path_info)
req.body_stream = request.body
req.content_type = request.content_type
req.content_length = request.content_length
http = Net::HTTP.new(#host, #port)
res = http.request(req)
I've tried several ways to add the proxy's parameters. But it seems nothing in Net::HTTP allows to add parameters to a body_stream request, only to a body one.
Is there a simpler way to proxy a rack request like that ? Or a clean way to add my parameters to my request ?
Well.. as i see it, this is a normal behaviour. I'll explain why. If you only have access to a Rack::Request,(i guess that) your middleware does not parse the response (you do not include something like ActionController::ParamsParser), so you don't have access to a hash of parameters, but to a StringIo. This StringIO corresponds to a stream like:
Content-Type: multipart/form-data; boundary=AaB03x
--AaB03x
Content-Disposition: form-data; name="param1"
value1
--AaB03x
Content-Disposition: form-data; name="files"; filename="file1.txt"
Content-Type: text/plain
... contents of file1.txt ...
--AaB03x--
What you are trying to do with the Net::HTTP class is to: (1). parse the request into a hash of parameters; (2). merge the parameters hash with your own parameters; (3). recreate the request. The problem is that Net::HTTP library can't do (1), since it is a client library, not a server one.
Therefore, you can not escape parsing some how your request before adding the new parameters.
Possible solutions:
Insert ActionController::ParamsParser before your middleware. After that, you may use the excellent rest-client lib to do something like:
RestClient.post ('http://your_server' + request.path_info), :params => params.merge(your_params)
You can attempt to make a wrapper on the StringIO object, and add, at the end of stream,your own parameters. However, this is not trivial nor advisable.
Might be one year too late, but I had the same issue verifying Paypal IPNs. I wanted to forward back the IPN request to Paypal for verification but needed to add :cmd => '_notify-validate'.
Instead of modifying the body stream, or body, I appended it as part of the URL path, like so:
reply_request = Net::HTTP::Post.new(url.path + '?cmd=_notify-validate')
It seems a bit of a hack, but I think it's worth it if you aren't going to use it for anything else.