we encounter some strange behaviour with our web application. Some POST requests do not have any http body, when they should. content-length is 0. There are no post parameters at all. We traced the network traffic at our loadbalancer and we see that we do not get any request body with some of our POST requests.
All broken POST requests have in common that they arrive via a proxy server.
We already found this question on SO:
Why "Content-Length: 0" in POST requests?
We are now using a frame escape javascript routine and it helps a bit. It seems that error rate drops. But we still have POST requests with no data which should never happen in our webapp. These requests does not come from hackers or alike.
Often we saw webwasher as a proxy. But most of the time we do not see which proxy is used.
In this PDF we saw a comment about missing POST parameters with webwasher
WebWasher - Transparent Authentication Guide
Notes on Some Pitfalls
Note that there are some pitfalls that must be taken into account when setting up transparent authentication:
POST requests will fail if the ICAP server sends an redirect to the authentication server. This affects, however, only the renewal of the mapping since for the browser the request was successful, and the POST body will not be sent again after the final redirect.
We would like to know if there is some workaround other than using only GET instead of POST.
We would also here if other sites had problems with missing POST data and which conclusion they made.
Are there any other reasons why POST data is not sent?
I've had issues with Microsoft's proxy server not playing well with web requests.
I've had to resort to forcing HTTP/1.0 and setting the KeepAlive property to false.
There's something about the way NTLM authentication works that causes the body to be sent sporadically.
I've added this to many of my web requests
protected override WebRequest GetWebRequest(Uri uri)
{
HttpWebRequest webRequest = (HttpWebRequest) base.GetWebRequest(uri);
webRequest.KeepAlive = false;
webRequest.ProtocolVersion=HttpVersion.Version10;
return webRequest;
}
Hope this helps!
Not really an answer, I guess, but I arrived here because we had a similar problem. Initially, we thought it was due to the clients being mobile, as this was a common theme, but we have now realised that the common denominator is proxies.
We now raise a http 400 when it happens.
Here are a few of the proxies, we've had issues with. Posting them to lead the casual googler here:
1.1 ACISA02S, 1.1 abc:3328 (squid/2.6.STABLE21)
1.1 ipcop00.cat.local:8000 (squid/2.6.STABLE21)
1.1 PRXTGLSRV01
1.1 ISA
No ones which conform to the spec:
Some HTTP methods MUST cause a cache to invalidate an entity.
...
POST
(the HTTP/1.0 spec states 'Applications must not cache responses to a POST request').
But there are is a LOT of badly written code out there.
What headers do you include in replies to POSTs on the URLs?
Related
I am running into an issue where my web code is invoking an ajax call to another endpoint on the same server that is not completing. I can see in the server logs that the request is processed and the response is sent, but the network tool shows the response as not finished (stalled) even after several minutes.
I am looking for more details on what is happening in the proxy (perhaps it is waiting on more data from the server, etc). But I cannot find any logging from the proxy to assist in debugging.
EDIT:
I have noticed this happens on all proxy requests with isXhr set.
OK, so to answer the original question regarding verbose logging, there isn't any. I had to sprinkle logging through the hammerhead code to get anything more meaningful than testcafe timing out on a selector.
Regarding the underlying issue (which would have been obvious long ago with decent proxy logging), I finally uncovered a node.js http module "Parse Error". While the browsers are more forgiving, node's http module will barf on a Content-Length header along with 'Transfer-Encoding : chunked'. Chunked encoding creats additional overhead and the Content-Length does not match the payload size any longer.
This would normally not have happened, but in our server implementation, the API calls are proxied and the far end headers were being passed on to the response leading to a collision in the headers.
Hopefully this will be useful for someone else.
You need to inspect the RequestPipeline module. It controls the procedures of sending the request, processing the response, and sending the modified response to the client.
I am newbie to website security and currently trying to understand Same-Origin-Policy in some depth.
While there are very good posts on stackoverflow and elsewhere about the concept of SOP, I could not find updated information on whether chrome and other browsers allow cross-domain XHR post requests to be 'sent' from the first place.
From this 5 year old post, it appears that chrome allows the request to pass through to the requested server but does not allow reading the response by the requester.
I tested that on my website trying to change user info on my server from a different domain. Details below:
My domain: "www.mysite.com"
Attacker domain: "www.attacker.mysite.com"
According to Same-Origin-Policy those two are considered different Origins.
User (while logged in to www.mysite.com) opens www.attacker.mysite.com and presses a button that fires a POST request to 'www.mysite.com' server...The submitted hidden form (without tokens in this case) has all the required information to change the user's info on 'www.mysite.com' server --> Result: CSRF successful attack: The user info does indeed change.
Now do the same but with javascript submitting the form through JQuery .post instead of submitting the form--> Result: Besides chrome giving the normal response:
No 'Access-Control-Allow-Origin' header is present on the requested
resource
, I found that no change is done on the server side...It seems that the request does not even pass through from the browser. The user info does not change at all! While that sounds good, I was expecting the opposite.
According to my understanding and the post linked above, for cross-domain requests, only the server response should be blocked by the browser not sending the post request to the server from the first place.
Also, I do not have any CORS configuration set; no Access-Control-Allow-Origin headers are sent. But even if I had that set, that should apply only on 'reading' the server response not actually sending the request...right?
I thought of preflights, where a request is sent to check if it's allowed on the server or not, and thus blocking the request before sending its actual data to change the user info. However, according to Access_Control_CORS , those preflights are only sent in specific situations which do not apply to my simple AJAX post request (which includes a simple form with enctype by default application/x-www-form-urlencoded and no custom headers are sent).
So is it that chrome has changed its security specs to prevent the post request to a cross domain from the first place?
or am I missing something here in my understanding to the same-origin-policy?
Either way, it would be helpful to know if there is a source for updated security measures implemented in different web browsers.
The XMLHttpRequest object behavior has been revisited with time.
The first AJAX request were unconstrained.
When SOP was introduced the XMLHttpRequest was updated to restrict every cross-origin request
If the origin of url is not same origin with the XMLHttpRequest origin the user agent should raise a SECURITY_ERR exception and terminate these steps.
From XMLHttpRequest Level 1, open method
The idea was that an AJAX request that couldn't read the response was useless and probably malicious, so they were forbidden.
So in general a cross-origin AJAX call would never make it to the server.
This API is now called XMLHttpRequest Level 1.
It turned out that SOP was in general too strict, before CORS was developed, Microsoft started to supply (and tried to standardize) a new XMLHttpRequest2 API that would allow only some specific requests, stripped by any cookie and most headers.
The standardization failed and was merged back into the XMLHttpRequest API after the advent of CORS. The behavior of Microsoft API was mostly retained but more complex (read: potentially dangerous) requests were allowed upon specific allowance from the server (through the use of pre-flights).
A POST request with non simple headers or Content-Type is considered complex, so it requires a pre-flight.
Pre-flights are done with the OPTIONS method and doesn't contain any form information, as such no updates on the server are done.
When the pre-flight fails, the user-agent (the browser) terminate the AJAX request, preserving the XMLHttpRequest Level 1 behavior.
So in short: For XMLHttpRequest the SOP was stronger, deny any cross-origin operations despite the goals stated by the SOP principles. This was possible because at the time that didn't break anything.
CORS loosened the policy allowing "non harmful" requests by default and allowing the negotiation of the others.
OK...I got it...It's neither a new policy in chrome nor missing something in SOP...
The session cookies of "www.mysite.com" were set to "HttpOnly" which means, as mentioned here, that they won't be sent along with AJAX requests, and thus the server won't change the user's details in point (4).
Once I added xhrFields: { withCredentials:true } to my post request, I was able to change the user's information in a cross-domain XHR POST call as expected.
Although this proves the already known fact that the browser actually sends the cross-domain post requests to the server and only blocks the server response, it might still be helpful to those trying to deepen their understanding to SOP and/or playing with CORS.
I've been working on a classic SPA where the front end app lives on app.example.com while the API lives on api.example.com, hence requiring the use of CORS requests. Have setup the server to return the CORS header, works fine.
Whenever an AJAX request is not simple, the browser makes an extra OPTIONS request to the server to determine if it can make the call with the payload. Find Simple Requests on MDN
The question is: What are the actual benefits of doing the OPTIONS request, especially in regards to security?
Some users of my app have significant geographical latency and since the preflight cache doesn't last long, the preflight requests cause latencies to be multiplied.
I'm hoping to make POST requests simple, but just embedding the Content-Type of application/json negates that. One potential solution is to "hack" it by using text/plain or encoding in the url. Hence, I hope to leave with a full understanding of what CORS preflight requests do for web security. Thanks.
As noted on the article you linked to:
These are the same kinds of cross-site requests that web content can
already issue, and no response data is released to the requester
unless the server sends an appropriate header. Therefore, sites that
prevent cross-site request forgery have nothing new to fear from HTTP
access control.
Basically it was done to make sure CORS does not introduce any extra means for cross-domain requests to be made that would otherwise be blocked without CORS.
For example, without CORS, the following form content types could only be done cross-domain via an actual <form> tag, and not by an AJAX request:
application/x-www-form-urlencoded
multipart/form-data
text/plain
Therefore any server receiving a request with one of the above content-types knows that there is a possibility of it coming from another domain and knows to take measures against attacks such as Cross Site Request Forgery. Other content types such as application/json could previously only be made from the same domain, therefore no extra protection was necessary.
Similarly requests with extra headers (e.g. X-Requested-With) would have previously been similarly protected as they could have only come from the same domain (a <form> tag cannot add extra headers, which was the only way previously to do a cross-domain POST). GET and POST are also the only methods supported by a form. HEAD is also listed here as it performs identically to GET, but without the message body being retrieved.
So, in a nutshell it will stop a "non simple" request from being made in the first place, without OPTIONS being invoked to ensure that both client and server are talking the CORS language. Remember that the Same Origin Policy only prevents reads from different origins, so the preflight mechanism is still needed to prevent writes from taking place - i.e. unsafe methods from being executed in a CSRF scenario.
You might be able to increase performance using the Access-Control-Max-Age header. Details here.
I'm loading my script on a domain and sending some data with POST and the use of Ext.Ajax.request() to that same domain.
Somehow the dev-tools show me, that there is a failed OPTIONS request.
Request URL : myurl-internal.com:8090/some/rest/api.php
Request Headers
Access-Control-Request-Headers : origin, x-requested-with, content-type
Access-Control-Request-Method : POST
Origin : http://myurl-internal.com:8090
It's both HTTP and not HTTPS. Same port, same host ... I don't know why it's doing this.
The server can't handle such stuff and so the request fails and the whole system stops working.
It's not really specific to Ext JS -- see these related threads across other frameworks. It's the server properly enforcing the CORS standard:
for HTTP request methods that can cause side-effects on user data (in
particular, for HTTP methods other than GET, or for POST usage with
certain MIME types), the specification mandates that browsers
“preflight” the request, soliciting supported methods from the server
with an HTTP OPTIONS request header, and then, upon “approval” from
the server, sending the actual request with the actual HTTP request
method.
If you're going to use CORS, you need to be able to either properly handle or ignore these requests on the server. Ext JS itself doesn't care about the OPTIONS requests -- you'll receive the responses as expected, but unless you do something with them they'll just be ignored (assuming the server actually allows whatever you're trying to do).
If you are NOT intending to use CORS (which sounds like you aren't purposefully going cross-domain) then you need to figure out why the server thinks the originating domain is different (I'm not sure about that). You could also bypass CORS altogether by using JsonP (via Ext's JsonP proxy).
Use relative url instead of absolute, then you will get expected result.
use before request
Ext.Ajax.useDefaultXhrHeader = false
A customer sometimes sends POST requests with Content-Length: 0 when submitting a form (10 to over 40 fields).
We tested it with different browsers and from different locations but couldn't reproduce the error. The customer is using Internet Explorer 7 and a proxy.
We asked them to let their system administrator see into the problem from their side. Running some tests without the proxy, etc..
In the meantime (half a year later and still no answer) I'm curious if somebody else knows of similar problems with a Content-Length: 0 request. Maybe from inside some Windows network with a special proxy for big companies.
Is there a known problem with Internet Explorer 7? With a proxy system? The Windows network itself?
Google only showed something in the context of NTLM (and such) authentication, but we aren't using this in the web application. Maybe it's in the way the proxy operates in the customer's network with Windows logins? (I'm no Windows expert. Just guessing.)
I have no further information about the infrastructure.
UPDATE: In December 2010 it was possible to inform one administrator about this, incl. links from the answers here. Contact was because of another problem which was caused by the proxy, too. No feedback since then. And the error messages are still there. I'm laughing to prevent me from crying.
UPDATE 2: This problem exists since mid 2008. Every few months the customer is annoyed and wants it to be fixed ASAP. We send them all the old e-mails again and ask them to contact their administrators to either fix it or run some further tests. In December 2010 we were able to send some information to 1 administrator. No feedback. Problem isn't fixed and we don't know if they even tried. And in May 2011 the customer writes again and wants this to be fixed. The same person who has all the information since 2008.
Thanks for all the answers. You helped a lot of people, as I can see from some comments here. Too bad the real world is this grotesque for me.
UPDATE 3: May 2012 and I was wondering why we hadn't received another demand to fix this (see UPDATE 2). Looked into the error protocol, which only reports this single error every time it happened (about 15 a day). It stopped end of January 2012. Nobody said anything. They must have done something with their network. Everything is OK now. From summer 2008 to January 2012. Too bad I can't tell you what they have done.
UPDATE 4: September 2015. The website had to collect some data and deliver it to the main website of the customer. There was an API with an account. Whenever there was a problem they contacted us, even if the problem was clearly on the other side. For a few weeks now we can't send them the data. The account isn't available anymore. They had a relaunch and I can't find the pages anymore that used the data of our site. The bug report isn't answered and nobody complaint. I guess they just ended this project.
UPDATE 5: March 2017. The API stopped working in the summer of 2015. The customer seems to continue paying for the site and is still accessing it in February 2017. I'm guessing they use it as an archive. They don't create or update any data anymore so this bug probably won't reemerge after the mysterious fix of January 2012. But this would be someone else's problem. I'm leaving.
Internet Explorer does not send form fields if they are posted from an authenticated site (NTLM) to a non-authenticated site (anonymous).
This is feature for challange-response situations (NTLM- or Kerberos- secured web sites) where IE can expect that the first POST request immediately leads to an HTTP 401 Authentication Required response (which includes a challenge), and only the second POST request (which includes the response to the challange) will actually be accepted. In these situations IE does not upload the possibly large request body with the first request for performance reasons. Thanks to EricLaw for posting that bit of information in the comments.
This behavior occurs every time an HTTP POST is made from a NTLM authenticated (i.e. Intranet) page to a non-authenticated (i.e. Internet) page, or if the non-authenticated page is part of a frameset, where the frameset page is authenticated.
The work-around is either to use a GET request as the form method, or to make sure the non-authenticated page is opened in a fresh tab/window (favorite/link target) without a partly authenticated frameset. As soon as the authentication model for the whole window is consistent, IE will start to send form contents again.
Definitely related: http://www.websina.com/bugzero/kb/browser-ie.html
Possibly related: KB923155
Full Explanation: IEInternals Blog – Challenge-Response Authentication and Zero-Length Posts
This is easy to reproduce with MS-IE and an NTLM authentication filter on server side. I have the same issue with JCIFS (1.2.), struts 1. and MS-IE 6/7 on XP-SP2. It was finally fixed. There are several workarounds to make it up.
change form method from POST (struts default setting) to GET.
For most pages with small sized forms, it works well. Unfortunately i have possibly more than 50 records to send in HTTP stream back to server side. IE has a GET URL limit 2038 Bytes (not parameter length, but the whole URL length). So this is a quick workaround but not applicable for me.
send a GET before POST action executing.
This was recommended in MS-KB. My project has many legacy procedures and i would not take the risk at the right time. I have never tried this because it still needs some extra authentication processing when GET is received by filter layer based on my understanding from MS-KB and I would not like to change the behavior with other browsers, e.g. Firefox, Opera.
detecting if POST was sent with zero content-length (you may get it from header properties hash structure with your framework).
If so, trigger an NTLM authentication cycle by get challenge code from DC or cache and expect an NTLM response.
When the NTLM type2 msg is received and the session is still valid, you don't really need to authenticate the user but just forward it to the expected action if POST content-length is not zero. BTW, this would increase the network traffics. So check your cache life time setting and SMB session soTimeOut configuration before applying the change plz.
Or, more simple, you may just send a 401-unauthorized status to MS-IE and the browser shall send back POST request with data in reply.
MS-KB has provided a hot-fix with KB-923155 (I could not post more than one link because of a low reputation number :{ ) , but it seems not working. Would someone post a workable hot-fix here? Thanks :) Here is a link for reference, http://www.websina.com/bugzero/kb/browser-ie.html
We have a customer on our system with exactly the same problem. We've pin pointed it down to the proxy/firewall. Microsoft's IAS. It's stripping the POST body and sending content-length: 0. Not a lot we can do to work around however, and down want to use GET requests as this exposes usernames/passwords etc on the URL string. There's nearly 7,000 users on our system and only one with the problem... also only one using Microsoft IAS, so it has to be this.
There's a good chance the problem is that the proxy server in between implements HTTP 1.0.
In HTTP 1.0 you must use the Content-Length header field: (See section 10.4 here)
A valid Content-Length is required on
all HTTP/1.0 POST requests. An
HTTP/1.0 server should respond with a
400 (bad request) message if it cannot
determine the length of the request
message's content.
The request going into the proxy is HTTP 1.1 and therefore does not need to use the Content-Length header field. The Content-Length header is usually used but not always. See the following excerpt from the HTTP 1.1 RFC S. 14.13.
Applications SHOULD use this field to
indicate the transfer-length of the
message-body, unless this is
prohibited by the rules in section
4.4.
Any Content-Length greater than or
equal to zero is a valid value.
Section 4.4 describes how to determine
the length of a message-body if a
Content-Length is not given.
So the proxy server does not see the Content-Length header, which it assumes is absolutely needed in HTTP 1.0 if there is a body. So it assumes 0 so that the request will eventually reach the server. Remember the proxy doesn't know the rules of the HTTP 1.1 spec, so it doesn't know how to handle the situation when there is no Content-Length header.
Are you 100% sure your request is specifying the Content-Length header? If it is using another means as defined in section 4.4 because it thinks the server is 1.1 (because it doesn't know about the 1.0 proxy in between) then you will have your described problem.
Perhaps you can use HTTP GET instead to bypass the problem.
This is a known problem for Internet explorer 6, but not for 7 that I know of. You can install this fix for the IE6 KB831167 fix.
You can read more about it here.
Some questions for you:
Do you know which type of proxy?
Do you know if there is an actual body sent in the request?
Does it happen consistently every time? Or only sometimes?
Is there any binary data sent in the request? Maybe the data starts with a \0 and the proxy has a bug with binary data.
If the user is going through an ISA proxy that uses NTLM authentication, then it sounds like this issue, which has a solution provided (a patch to the ISA proxy)
http://support.microsoft.com/kb/942638POST requests that do not have a POST body may be sent to a Web server that is published in ISA Server 2006
I also had a problem where requests from a customer's IE 11 browser had Content-Length: 0 and did not include the expected POST content. When the customer used Firefox, or Chrome the expected content was included in the request.
I worked out the cause was the customer was using a HTTP URL instead of a HTTPS URL (e.g. http://..., not https://...) and our application uses HSTS. It seems there might be a bug in IE 11 that when a request gets upgraded to HTTPS due to HSTS the request content gets lost.
Getting the customer to correct the URL to https://... resulted in the content being included in the POST request and resolved the problem.
I haven't investigated whether it is actually a bug in IE 11 any further at this stage.
Are you sure these requests are coming from a "customer"?
I've had this issue with bots before; they sometimes probe sites for "contact us" forms by sending blank POST requests based on the action URI in FORM tags they discover during crawling.
Presence and possible values of the ContentLength header in HTTP are described in the HTTP ( I assume 1/1) RFC:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13
In HTTP, it SHOULD be sent whenever the message's length can be determined prior to being transferred
See also:
If a message is received with both a
Transfer-Encoding header field and a Content-Length header field,
the latter MUST be ignored.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4
Maybe your message is carrying a Transfer-Encoding header?
Later edit: also please note "SHOULD" as used in the RFC is very important and not equivalent to "MUST":
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
Ref: http://www.ietf.org/rfc/rfc2119.txt
We had a customer using same website in anonymous and NTLM mode (on different ports). We found out that in our case the 401 was related to Riverbed Steelhead application used for http optimization. The first signal pointing us into that direction was a X-RBT-Optimized-By header. The issue was the Gratuitous 401 feature:
This feature can be used with both per-request and per-connection
authentication but it‘s most effective when used with per-request
authentication. With per-request authentication, every request must be
authenticated against the server before the server would serve the
object to the client. However, most browsers do not cache the server‘s
response requiring authentication and hence it will waste one
round-trip for every GET request. With Gratuitous 401, the client-side
Steelhead appliance will cache the server response and when the client
sends the GET request without any authentication headers, it will
locally respond with a ―401 Unauthorized‖ message and therefore saving
a round trip. Note that the HTTP module does not participate in the
actual authentication itself. What the HTTP module does is to inform
the client that the server requires authentication without requiring
it to waste one round trip.
Google also shows this as an IE (some versions, anyway) bug after an https connection hits the keepalive timeout and reconnects to the server. The solution seems to be configuring the server to not use keepalive for IE under https.
Microsoft's hotfix for KB821814 can set Content-Length to 0:
The hotfix that this article describes implements a code change in Wininet.dll to:
Detect the RESET condition on a POST request.
Save the data that is to be posted.
Retry the POST request with the content length set to 0. This prevents the reset from occurring and permits the authentication process to complete.
Retry the original POST request.
curl sends PUT/POST requests with Content-Length: 0 when configured to use HTTP proxy. It's trick to overcome required buffering in case of first unauthorized PUT/POST request to proxy. In case of GET/HEAD requests curl simply repeats the query. The scheme for PUT/POST is like:
Send first PUT/POST request with Content-Length set to 0.
Get answer. HTTP status code of 407 means we have to use proxy
authorization. Prepare headers for proxy authentication for send request.
Send request again with filled headers for proxy authentication and real data to POST/PUT.