how to catch "http: server closed idle connection" error - go

In my go application I am getting the following error: "http: server closed idle connection". I would like to catch it and retry my http connection if it's encountered.
I found that this error comes from the "net/http" package and furthermore from the transport implementation. In particular it's defined here
I get it wrapped in url.Error but this is all I was able to find out. Do you know how I can actually catch this error?
Edit:
I am using elastic search client, which in turn is using the net/http. From the client I get the above mentioned error and would like to retry my elastic search request as being transient. For now the way I am catching transient errors is:
if urlErr, ok := err.(*url.Error); ok && (urlErr.Temporary() || urlErr.Err == io.EOF) {
return retryRequest()
}

Comments in net/http/transport.go say this (located here):
if err == errServerClosedIdle {
// The server replied with io.EOF while we were trying to
// read the response. Probably an unfortunately keep-alive
// timeout, just as the client was writing a request.
return true
}
Go will attempt to retry the request if it finds a non-nil http.Request.GetBody (found here), so I think it is expected to retry the request (or provide a GetBody function).
As for the leading error cause, you might want to check server's keep-alive functionality, my guess is that server is sending TCP connection reset (interpreted as io.EOF) prematurely. You might want to try disabling keep-alives and see if that changes anything.

As Gray Fox mentioned, Go will usually retry these requests. However, this is not the case for requests that aren't "idempotent". I was running into a few of these errors on POST requests, and after searching around I found this Github issue that says:
The HTTP Transport would normally retry those requests, but because they're POST requests they're not idempotent and Go conservatively assumes it might be unsafe to retry them.
Later in the issue someone mentioned that many banking APIs use some variant of the Idempotent-Key header to make them "safe". Stripe is a company that uses that header, and this is what their API docs say about it:
The API supports idempotency for safely retrying requests without accidentally performing the same operation twice.
So there you go, that's what the header is for.
In short, if you want Go's HTTP Transport to retry POST requests (and other non-idempotent requests), you need to include either the Idempotency-Key or X-Idempotency-Key header. That doesn't necessarily make those requests idempotent, but it will make them retry-able from net/http's perspective.
Links:
Source code check for the Idempotency-Key headers - https://github.com/golang/go/blob/2cd2ff6f564dce5be0c4fb7f06338ff7af3fc9a9/src/net/http/request.go#L1401)
Github issue where I found this information: https://github.com/golang/go/issues/19943

Related

Can I make DataPower NOT return http 500 for failed WSP activation attempts?

Right now, a request sent to a valid DataPower port with an invalid URI is answered with http 500.
This has been deemed to be a security risk, and I am looking for a way to reconfigure it so that it instead causes a connection termination, or anything that wouldn't inform a possible attacker that they have found an active port.
For example: if a valid WSP URL is http://datapower:80/abc, I would like someone who tries http://datapower:80**/a** to get the same result as someone who tries http://datapower**:90/**abc.
The only way I know of to return a custom error code is through a gateway script in a Policy object, which is obviously irrelevant to wrong URI calls that don't activate a WSP/MPGW.
Is there any way to arrange this in DataPower?

How can I get more verbose debugging information from hammerhead?

I am running into an issue where my web code is invoking an ajax call to another endpoint on the same server that is not completing. I can see in the server logs that the request is processed and the response is sent, but the network tool shows the response as not finished (stalled) even after several minutes.
I am looking for more details on what is happening in the proxy (perhaps it is waiting on more data from the server, etc). But I cannot find any logging from the proxy to assist in debugging.
EDIT:
I have noticed this happens on all proxy requests with isXhr set.
OK, so to answer the original question regarding verbose logging, there isn't any. I had to sprinkle logging through the hammerhead code to get anything more meaningful than testcafe timing out on a selector.
Regarding the underlying issue (which would have been obvious long ago with decent proxy logging), I finally uncovered a node.js http module "Parse Error". While the browsers are more forgiving, node's http module will barf on a Content-Length header along with 'Transfer-Encoding : chunked'. Chunked encoding creats additional overhead and the Content-Length does not match the payload size any longer.
This would normally not have happened, but in our server implementation, the API calls are proxied and the far end headers were being passed on to the response leading to a collision in the headers.
Hopefully this will be useful for someone else.
You need to inspect the RequestPipeline module. It controls the procedures of sending the request, processing the response, and sending the modified response to the client.

Response code: 503 Response message: Service Unavailable

I have given two url in csv file and trying to load that in jmeter. I am getting 503 response code.
As per HTTP 503 status code description
The HyperText Transfer Protocol (HTTP) 503 Service Unavailable server error response code indicates that the server is not ready to handle the request.
Common causes are a server that is down for maintenance or that is overloaded. This response should be used for temporary conditions and the Retry-After HTTP header should, if possible, contain the estimated time for the recovery of the service.
Note: together with this response, a user-friendly page explaining the problem should be sent.
Caching-related headers that are sent along with this response should be taken care of, as a 503 status is often a temporary condition and responses shouldn't usually be cached.
So I can think of at least 3 possible issues:
Your server is really overloaded, you can try checking Response Data tab of the View Results Tree listener - if you're lucky - you will the the aforementioned "user-friendly page explaining the problem"
Your request is malformed and server is sending 503 response instead of 4xx which indicates client-side error. Double check what URL(s) you're trying to open by looking into Request tab of the View Results Tree listener and try to open it in the browser. If you're building the URL from parts (i.e. host, port, protocol, query string, etc.) - you might want to check if the values are correctly read from the CSV file using Debug Sampler
I fail to see HTTP Header Manager in your Test Plan, some servers check the client Headers, for example User-Agent or Accept and may report an error if the required header is missing (i.e. indicating that the browser is not supported)

JMeter error : java.net.SocketException: Unexpected end of file from server

I have read and tried the solutions shared in the previous posts for this error but nothing helped me to fix this. Kindly help.
I am making a HTTPS API call. A very simple call which accepts a 2 KB JSON message via POST method and sends a one word acknowledgement. It works perfectly fine in Postman tool. In JMeter, no. of threads (users) is kept as 25. It works perfectly fine one time with all 25 success response and at times getting few failure response as
Response code: Non HTTP response code: java.net.SocketException
Response message: Non HTTP response message: Unexpected end of file from server
No consistency at all. I have also tried with both enabling and disabling "use keepalive" checkbox. Both giving me all success one time ; few/all failure at another time with the above error. Please help. Thank you.
Below are the JMeter settings:
HTTP Header Manager : Content-Type - application/json
HTTPRequest sampler : protocol - HTTPS
Server Name or IP : project server name
Method : POST
PATH : The required path with https authentication details passed as parameters
IMPLEMENTATION : Set to JAVA (HTTPCLIENT4 was giving me “443 failed to respond” error)
Add header
Connection Keep-Alive
Both errors (Java implementation and Apache HTTPClient4 implementation), are essentially saying the same thing: The server closed the connection, without providing any response. I think there could be the following reasons:
Authentication problem. If server side checks authentication before processing a request (e.g. using Spring), it may be rejecting your request, it may be not bothering with any response if authentication is not considered correct.
Request Issue. Some less noticeable properties of the request you send via Jmeter are different from what you send in Postman. It could be some minor thing with formatting, or some headers server expects. Some of such inconsistencies can also cause Load Balancer (if you use one) to reject request before it's delivered to the server.
Certificate issues. Since you are using HTTPS, you need to make sure your certificate is setup correctly on JMeter side.
So I suggest:
Review server logs and see if your request makes it to the server. If yes, you might be able to see how it was rejected. If not, you need to trace back and see who rejected it (LB, authentication, etc)
Compare headers and body sent by JMeter vs Postman line to line (use TCPDump for example to obtain it).

Missing POST Parameters with proxy servers

we encounter some strange behaviour with our web application. Some POST requests do not have any http body, when they should. content-length is 0. There are no post parameters at all. We traced the network traffic at our loadbalancer and we see that we do not get any request body with some of our POST requests.
All broken POST requests have in common that they arrive via a proxy server.
We already found this question on SO:
Why "Content-Length: 0" in POST requests?
We are now using a frame escape javascript routine and it helps a bit. It seems that error rate drops. But we still have POST requests with no data which should never happen in our webapp. These requests does not come from hackers or alike.
Often we saw webwasher as a proxy. But most of the time we do not see which proxy is used.
In this PDF we saw a comment about missing POST parameters with webwasher
WebWasher - Transparent Authentication Guide
Notes on Some Pitfalls
Note that there are some pitfalls that must be taken into account when setting up transparent authentication:
POST requests will fail if the ICAP server sends an redirect to the authentication server. This affects, however, only the renewal of the mapping since for the browser the request was successful, and the POST body will not be sent again after the final redirect.
We would like to know if there is some workaround other than using only GET instead of POST.
We would also here if other sites had problems with missing POST data and which conclusion they made.
Are there any other reasons why POST data is not sent?
I've had issues with Microsoft's proxy server not playing well with web requests.
I've had to resort to forcing HTTP/1.0 and setting the KeepAlive property to false.
There's something about the way NTLM authentication works that causes the body to be sent sporadically.
I've added this to many of my web requests
protected override WebRequest GetWebRequest(Uri uri)
{
HttpWebRequest webRequest = (HttpWebRequest) base.GetWebRequest(uri);
webRequest.KeepAlive = false;
webRequest.ProtocolVersion=HttpVersion.Version10;
return webRequest;
}
Hope this helps!
Not really an answer, I guess, but I arrived here because we had a similar problem. Initially, we thought it was due to the clients being mobile, as this was a common theme, but we have now realised that the common denominator is proxies.
We now raise a http 400 when it happens.
Here are a few of the proxies, we've had issues with. Posting them to lead the casual googler here:
1.1 ACISA02S, 1.1 abc:3328 (squid/2.6.STABLE21)
1.1 ipcop00.cat.local:8000 (squid/2.6.STABLE21)
1.1 PRXTGLSRV01
1.1 ISA
No ones which conform to the spec:
Some HTTP methods MUST cause a cache to invalidate an entity.
...
POST
(the HTTP/1.0 spec states 'Applications must not cache responses to a POST request').
But there are is a LOT of badly written code out there.
What headers do you include in replies to POSTs on the URLs?

Resources