Many libraries include Expect: 100-continue on all HTTP 1.1 POST and PUT requests by default.
I intend to reduce perceived latency by removing 100-continue mechanism on the client side on those requests for which I know the expense of sending data right away is less than waiting a roundtrip for 100-continue, namely on short requests.
Of course I still want all the other great features of HTTP 1.1, thus only I want to kill Expect: 100-continue header. I have two options:
remove expect header entirely, or
send empty expect header, Expect:\r\n
Is there ever any difference between the two?
Any software that might break for one or the other?
Nothing should break if you remove the Expect header, but I know that Microsoft IIS has had issues with 100 Continue in the past. For example, IIS5 always sends 100 continue responses. So, I wonder if at least some of the uses of it in libraries might be to work around similarly broken behaviour in servers.
Many libraries seem to set this header and then not actually handle 100 Continue properly - e.g. they begin to send the request body immediately without waiting for a 100 Continue and then don't handle the fact that the server might send back any HTTP error code before they've finished sending the request body (the first part's OK, it's the second part which is broken - see later in my answer). This leads me to believe that some authors have just copied it from elsewhere without fully understanding the subtleties.
I can't see any reason to include a blank Expect header - if you're not going to include 100-continue (or some other Expect clause) then omit the header entirely. The only reason to include it would be to work around broken webservers, but I'm not aware of any which behave in this way.
Finally, if you're just looking to reduce roundtrip latencies it seems to me that it wouldn't actually be inconsistent with the RFC to simply begin to transmit the request body immediately. You're not supposed to wait indefinitely to send the request body (as per the RFC), so you're behaving to the spec - it's just your timeout before sending anyway is zero.
You must be aware that servers are at liberty to not send the 100 Continue response if they've already received some of the request body, so you have to handle servers which send 100 Continue, those which send nothing and wait for the full request and those which immediately send any HTTP error code (which may be 417, but more likely a generic 4xx code). In this way, your short requests shouldn't have any overhead (aside from the Expect header) but you won't have to wait for the 100 Continue. Of course, for this approach to work you'll need to be doing things in a way which lets you interrupt the request as soon as the server returns an error code (e.g. non-blocking IO with poll() or select()).
Doing things this way might help keep your code more consistent between small and large requests while reducing the latency. The downside is that it's perhaps not what the RFC authors had in mind, even if it doesn't explicitly violate any of the requirements. Also, it might make your later code more complicated if you're not already doing non-blocking IO or similar.
Related
So from my understanding, HTTP requests can be provided responses before the server is even done reading the complete request, such as in this case:
Upon receiving a request which includes an Expect request-header
field with the "100-continue" expectation, an origin server MUST
either respond with 100 (Continue) status and continue to read
from the input stream, or respond with a final status code. The
origin server MUST NOT wait for the request body before sending
the 100 (Continue) response.
This means that, theoretically, the server can read just the protocol/version number and maybe the first header or two and determine whether or not to continue reading the request. Or it can take just that first bit of information and start sending a response immediately.
Well, I am in love with PEG grammars for their ease of understanding and simplicity to construct. However, a PEG grammar parses using recursive decent from what I have learned, which means it goes like "is this an HTTP request?... is this an HTTP header list?... is this an HTTP header?... is this an HTTP header name?... is this a double quote?... etc." It might get done with the header list and then move onto the body, but it is still in the context of the HTTP request as a whole. So it's only done after the whole request is parsed.
My question is how can you use a grammar defined like a PEG, but in a different way somehow so it can handle this streaming situation? Would it simply make sense to emit events during parsing, so when it is done with each Header it emits that header? But then what if the body is malformed, then the whole parse will fail, and so we'd need to say "cancel our earlier header handling code". But wouldn't you need to do this anyways with a non-recursive-descent parser, where say you respond with the "continue" stuff initially, but then it fails, you then rewrite the status to say "error 404" or whatever. How does that work? And how can I conceptualize how the grammar should be used in a parser to handle this streaming case?
Side note: I do not want to write a custom manual HTTP parser. Not because it's hard but because it's ugly. The requirements for this project are that the HTTP parsing be done using an HTTP grammar (of at least some sort).
I'm coming to this from the InfoSec side, not the AppDev side, I just wanted to put that caveat in first. The issue is that my WAF is blocking certain images with the response, HTTP protocol compliance failed:Body in GET or HEAD requests. I need to justify keeping this rule active, so I'm asking, as a non-developer:
is this getting blocked because that's the rules of GET and HEAD requests, or can we allow Body in GET and HEAD requests, but it's really not a good idea ?
Why is it not a good idea? What are the potential problems that arise from allowing Body in a GET or HEAD request?
Thanks in advance for everyone's help.
GET and HEAD don't send request bodies, only POST and PUT do (see RFC 2616)
You are blocked for another reason, as I saw myself often, one of the request headers Content-Length or Transfer-Encoding which can be tolerated if really there's no request body at all (Content-Length: 0). As the WAF finds these headers, it considers the request as having a body, even of size 0.
If you loosen the policy, you will allow legitimate traffic but also open the door to abnormal traffic on GET/PUT. To circumvent this, you can add an iRule or LTM policy to remove the headers on GET/PUT, until F5 releases a better version of the software to not block the traffic when the body is of size 0.
The potential problem comes when a bugged Web server would buffer the data sent in a GET/HEAD body instead of returning a 400 error, and ignoring the data. This data could lead to memory consumption, or to injecting hacker's data into legitimate users requests with unknown results at this time. If you are confident in your Web server, you may loosen the WAF policy.
I have to ask a plaintive question. I know that http is normally request-response. Can it be request-done?
We have a situation where we would like to send an ajax call off to one server, and then when that completes post a form to another server. We can't send them both without coordinating them, because the post makes the browser navigate to another server, and we lose our context.
When I am currently doing is to do the first ajax call, and then in its callback, I'm doing document['order-form'].submit(). My boss pointed out that if the ajax call isn't completed for a while, the user will see his browser not make progress, even though it's still responsive. He wanted me to put a reasonable timeout on the ajax call.
But really, the ajax call is a "nice but not necessary" thing. My boss would be equally happy if we could send it and forget about it.
I'm having a lot of trouble formulating an appropriate query for Google. "Use HTTP like UDP" doesn't work. A lot of things don't work. Time to ask a human.
If you look at the ISO-OSI model of networking, HTTP is an application layer protocol and UDP is in the transport layer. HTTP typically uses TCP and rarely uses UDP. RTP (Realtime Transport Protocol) however uses UDP and is used for media streaming. Here is one more thing, UDP is not going to assure you a 100% transport, whereas TCP tries to (when packet loss is detected, TCP tries a re-transmission). So we expect drops in UDP. So when you say - fire and forget - What happens when your packet fails to reach?
So I guess you got confused between UDP and HTTP (and I am sorry if that' s not the case and there is something really with HTTP using UDP for web pages since I am not aware of it right now)
The best way, IMHO, to co-ordinate an asynchronous process like this is to have an AJAX call (with CORS enabled if required) like what you have written currently, coupled with good UI/UX frontends which intelligently shows progress/status to the end user.
Also - maybe we could tune up the process which makes the AJAX response slower..say a DB call which is supposed to return data can be tuned up a bit.
Here's what Eric Bidelman says:
// Listen to the upload progress.
var progressBar = document.querySelector('progress');
xhr.upload.onprogress = function(e) {
if (e.lengthComputable) {
progressBar.value = (e.loaded / e.total) * 100;
progressBar.textContent = progressBar.value; // Fallback for unsupported browsers.
}
};
I think this has the germ of an answer. 1) We can find out when the request has entirely gone. 2) We can choose not to have handlers for the response.
As soon as you have been informed that the request has gone out, you can take your next step, including navigating to another page.
I'm not sure, however, how many browsers support xhr.upload.onprogress.
If something is worth doing, surely it's worth knowing whether what you requested was done or not. Otherwise how can you debug problems, or give any kind of good user experience?
A response is any kind of response, it need not carry a message body. A simple 204 response could indicate that something succeeded, as opposed to a 403 or 401 which may require some more action.
I think I've figured out the answer. And it is extremely simple. Good across all browsers.
Just add xhr.timeout = 100; to your ajax call. If it takes the server a full second to respond, you don't care. You already moved on at 1/10 second.
So in my case, I put document['order-form'].submit() in my timeout handler. When the browser navigates away, I am assured that the request has finished going out.
Doesn't use any esoteric knowledge of protocols, or any recent innovations.
I have a set of resources whose representations are lazily created. The computation to construct these representations can take anywhere from a few milliseconds to a few hours, depending on server load, the specific resource, and the phase of the moon.
The first GET request received for the resource starts the computation on the server. If the computation completes within a few seconds, the computed representation is returned. Otherwise, a 202 "Accepted" status code is returned, and the client must poll the resource until the final representation is available.
The reason for this behavior is the following: If a result is available within a few seconds, it needs to be retrieved as soon as possible; otherwise, when it becomes available is not important.
Due to limited memory and the sheer volume of requests, neither NIO nor long polling is an option (i.e. I can't keep nearly enough connections open, nor even can I even fit all of the requests in memory; once "a few seconds" have passed, I persist the excess requests). Likewise, client limitations are such that they cannot handle a completion callback, instead. Finally, note I'm not interested in creating a "factory" resource that one POSTs to, as the extra roundtrips mean we fail the piecewise realtime constraint more than is desired (moreover, it's extra complexity; also, this is a resource that would benefit from caching).
I imagine there is some controversy over returning a 202 "Accepted" status code in response to a GET request, seeing as I've never seen it in practice, and its most intuitive use is in response to unsafe methods, but I've never found anything specifically discouraging it. Moreover, am I not preserving both safety and idempotency?
So, what do folks think about this approach?
EDIT: I should mention this is for a so-called business web API--not for browsers.
If it's for a well-defined and -documented API, 202 sounds exactly right for what's happening.
If it's for the public Internet, I would be too worried about client compatibility. I've seen so many if (status == 200) hard-coded.... In that case, I would return a 200.
Also, the RFC makes no indication that using 202 for a GET request is wrong, while it makes clear distinctions in other code descriptions (e.g. 200).
The request has been accepted for processing, but the processing has not been completed.
We did this for a recent application, a client (custom application, not a browser) POST'ed a query and the server would return 202 with a URI to the "job" being posted - the client would use that URI to poll for the result - this seems to fit nicely with what was being done.
The most important thing here is anyway to document how your service/API works, and what a response of 202 means.
From what I can recall - GET is supposed to return a resource without modifying the server. Maybe activity will be logged or what have you, but the request should be rerunnable with the same result.
POST on the other hand is a request to change the state of something on the server. Insert a record, delete a record, run a job, something like that. 202 would be appropriate for a POST that returned but isn't finished, but not really a GET request.
It's all very puritan and not well practiced in the wild, so you're probably safe by returning 202. GET should return 200. POST can return 200 if it finished or 202 if it's not done.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
In case of a resource that is supposed to have a representation of an entity that is clearly specified by an ID (as opposed to a "factory" resource, as described in the question), I recommend staying with the GET method and, in a situation when the entity/representation is not available because of lazy-creation or any other temporary situation, use the 503 Service Unavailable response code that is more appropriate and was actually designed for situations like this one.
Reasoning for this can be found in the RFCs for HTTP itself (please verify the description of the 503 response code), as well as on numerous other resources.
Please compare with HTTP status code for temporarily unavailable pages. Although that question is about a different use case, it actually relates to the exact same feature of HTTP.
I've wrote a SIP UAC, and I've tried a few ways to detect and ignore repeating incoming messages from the UAS, but with every approach I tried, something went wrong, my problem is that all the messages that has to do with the same call has the same signature, and to compare all of the message text is too much, so I was wondering, what parameter that compose a message should I be looking at when trying to detect these repeating messages.
UPDATE:
I had a problem with an incoming Options, which I handled with sending the server an empty Ok response. (Update: after a while of testing I noticed, that I still get every now and then I get another Options request, few every few second, so I try responding with a Bad request, and now I only get the Options request once/twice every registration/reregistration)
currently I have repeating messages of SessionInPogress, and different error messages such as busy here, and unavailable, I get so many of these, and it messes my log up, I would like to filter them.
any idea how to achieve that?
UPDATE:
I'll try your Technics before posting back, perhaps this would solve my problems
Here is what I used, it works nicely:
private boolean compare(SIPMessage message1, SIPMessage message2) {
if (message1.getClass() != message2.getClass())
return false;
if (message1.getCSeq().getSeqNumber() != message2.getCSeq().getSeqNumber())
return false;
if (!message1.getCSeq().getMethod().equals(message2.getCSeq().getMethod()))
return false;
if (!message1.getCallId().equals(message2.getCallId()))
return false;
if (message1.getClass()==SIPResponse.class)
if(((SIPResponse)message1).getStatusCode()!=((SIPResponse)message2).getStatusCode())
return false;
return true;
}
Thanks,
Adam.
It's a bit more complicated than ChrisW's answer.
First, the transaction layer filters out most retransmissions. It does this by, for most things, comparing the received message against a list of current transactions. If a transaction's found, that transaction will mostly swallow retransmissions as per the diagrams in RFC 3261, section 17. For instance, a UAC INVITE transaction in the Proceeding state will drop a delayed retransmitted INVITE.
Matching takes place in one of two ways, depending on the remote stack. If it's an RFC 3261 stack (the branch parameter on the topmost Via starts with "z9hG4bK") then things are fairly straightforward. Section 17.2.3 covers the full details.
Matching like this will filter out duplicate/retransmitted OPTIONS (which you mention as a particular problem). OPTIONS messages don't form dialogs, so looking at CSeq won't work. In particular, if the UAS sends out five OPTIONS requests which aren't just retransmissions, you'll get five OPTIONS requests (and five non-INVITE server transactions).
Retransmitted provisional responses to a non-INVITE transaction are passed up to the Transaction-User layer, or core as it's sometimes called, but other than the first one, final responses are not. (Again, you get this simply by implementing the FSM for that transaction - a final response puts a UAC non-INVITE transaction in the Completed state, which drops any further responses.
After that, the Transaction-User layer will typically receive multiple responses for INVITE transactions.
It's perfectly normal for a UAS to send multiple 183s, at least for an INVITE. For instance it might immediately send a 100 to quench your retransmissions (over unreliable transports at least), then a few 183s, a 180, maybe some more 183s, and finally a 200 (or more, for unreliable transports).
It's important that the transaction layer hands up all these responses because proxies and user agents handle the responses differently.
At this level the responses aren't, in a way, retransmitted. I should say: a UAS doesn't use retransmission logic to send loads of provisional responses (unless it implements RFC 3262). 200 OKs to INVITEs are resent because they destroy the UAC transaction. You can avoid their retransmission by sending your ACKs timeously.
I think that a message is duplicate/identical, if its ...
Cseq
Call-ID
and method name (e.g. "INVITE")
... values match that of another message.
Note that a response message has the same CSeq as the request to which it's responding; and, that a single request you get several, provisional, but non-duplicate responses (e.g. RINGING followed by OK).