In my production environment I got the following error in my server:
Cannot forward to error page for request [/api/validation] as the response has already been committed. As a result, the response may have the wrong status code. If your application is running on WebSphere Application Server you may be able to resolve this problem by setting com.ibm.ws.webcontainer.invokeFlushAfterService to false
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Connection reset by peer
Now I created a client and produced 1000 thread every second to call this [/api/validation].
The error I got was
Exception in thread "Thread-9954" org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://localhost:7080/v1/name/validate": Timeout waiting for connection from pool; nested exception is org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool.
Now I want to know is what is the cause of Connection reset by peer .
According to what I know is this error occurs when the client aborts the connection by sending the RST packet.
I set the socket Timeout of my client's rest template to 9000. I make the server sleep for about 15000 MS. Now shouldn't the server get Connection reset by peer as the server tries to send the response after 15 seconds and my client just waits for about 9 seconds. Shouldn't I get the error?
Also in the production environment the wait time (Rest template socket time out) for the client is set to about a 90 seconds ( more than the time the server requires to response). Why is the error being produced in the production?
Related
I am getting socket exception for broken pipe in my client side.
[write] I/O error: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Broken pipe (Write failed)
[LoggingManagedHttpClientConnection::shutdown] http-outgoing-278: Shutdown connection
1520546494584[20180308 23:01:34] [ConnectionHolder::abortConnection] Connection discarded
1520546494584[20180308 23:01:34] [BasicHttpClientConnectionManager::releaseConnection] Releasing connection [Not bound]
It seems that the upgradation of httpclient jar is causing issue.
Issue is not coming with httpclient-4.3.2
Exception is coming in every 2 minutes. Issue is intermittent at times.
after , send expect:100-continue ,conn.flush is throwing exception
client and server are Linux machine
client uses http jar to make request to server REST.
Please help me in debugging the issue
can httpjar cause such issue?
The persistent connections that are kept alive by the connection manager become stale. That is, the target server shuts down the connection on its end without HttpClient being able to react to that event, while the connection is being idle, thus rendering the connection half-closed or 'stale'
This is a general limitation of the blocking I/O in Java. There is simply no way of finding out whether or not the opposite endpoint has closed connection other than by attempting to read from the socket.
If a stale connection is used to transmit a request message the request execution usually fails in the write operation with SocketException and gets automatically retried.
Apache HttpClient works this problem around by employing the so stale connection check which is essentially a very brief read operation. However, the check can and often is disabled. In fact it is often advisable to have it disabled due to extra latency the check introduces.
The handling of stale connections was changed in version 4.4. Previously, the code would check every connection by default before re-using it. The code now only checks the connection if the elapsed time since the last use of the connection exceeds the timeout that has been set. The default timeout is set to 2000ms
I have 50 threads test in JMeter with multiple session but when I test it half of the threads is failed and I got this error Connection:
Response code: 500
Response message: Connection refused: connect Aborting action - session 656255658 was closed
Check 2 things:
are you sure you’re not reusing same session accross threads ? Are you correctly correlating the session id.
If issue only happens over some limit (not at 25 users for example, but at 50) then it’s a load issue or configuration limit on server side
I have increased the concurrent tasks to be '10' for PutSQL processor.
At that time it shows below error but there is no data loss.
failed to process due to org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object; rolling back session:
if i have remove concurrent tasks then it worked without those exception
while google this exception i have found answer in below link
I am getting Cannot get a connection, pool error Timeout waiting for idle object, When I try to create more than 250 threads in my web application
But i don't know how to avoid this issue in NiFi putSQL.
Can anyone help me to resolve this?
This exception occurs when the pool manager cannot produce a viable connection to a waiting requester and the maxWait has passed therefore triggering a timeout.
There are several causes, but they usually fall into 2 main categories:
The DB is down or unreachable.
The connection pool (which is set to 100 max active) is out of connections.
DBCPConnectionPool controller service in nifi has 8 max connection by default and 500 milli seconds max wait time. When PutSQL processor occupied 8 connection from DBCP pool and when it request for 9th connection or exceed the max connection limit then it will throw "Cannot get a connection" exception.
You can try 2 things to avoid this Exception :
You can increase the "Max Wait Time" in DBCPConnectionPool controller
service configuration.
You can increase the "Max Total Connections" limit in
DBCPConnectionPool controller service configuration.
Kindly find the below screenshot where you need to do changes.
It might resolve your issue.
This exception can occurs if some connections are never closed so they do not become available in the pool again.
So more and more connections remain open until reaching the max.
Make sure all threads are closing the connections used.
I am getting 'Non HTTP response message: Connection timed out: connect' for some HTTP requests so I tried to set the connection/response timeout value to 2 minutes (which is more than the connect time required for failing HTTP requests). To do this, I updated "HTTP Request Defaults" and added 120000 as Connect and Response Timeouts.
HTTP Request Defaults timeouts
[
However, when I run the test again, the HTTP requests still gave the same error. The sample result is as follows -
Load time: 21007
Connect Time: 21007
Latency: 0
Size in bytes: 2212
Sent bytes:0
Headers size in bytes: 0
Body size in bytes: 2212
Sample Count: 1
Error Count: 1
Data type ("text"|"bin"|""): text
Response code: Non HTTP response code: java.net.ConnectException
Response message: Non HTTP response message: Connection timed out: connect
It looks like the timeout value I set in HTTP request Defaults is not getting used here. I also tried to set the value of httpclient.timeout=120000 in jmeter.properties but no change. Have I missed something?
Can somebody please help me with this?
Thanks.
Edit - I have multiple HTTP requests and each run, different requests time-out. Here is one of the HTTP requests -
Updates:
I tried changing the Timeouts values in HTTP Request Defaults to very low (2000) to see how HTTP requests work. In this case, I was getting different error for requests exceeding connection time of 2000ms -
Non HTTP response code: org.apache.http.NoHttpResponseException/Non HTTP response message: : failed to respond
So I think changing the timeout values is not affecting my original error -
Non HTTP response code: java.net.ConnectException/Non HTTP response message: Connection timed out: connect
What is the difference between these two message?
The issue seems more of a server configuration of connection timeout than client side configuration of connection timeout, though both must be configured appropriately.
Default connectionTimeout in tomcat server is 20 seconds. and you request is failed due to connection timeout at 21 seconds. so, though you configured at client side (120000) you must configure appropriately at server side as well, otherwise, server forces to close the connection attempt and raises Connect Timeout exception.
Reference:
The HTTP Connector (refer connectionTimeout attribute)
Recently I have faced the same problem and found that it is the default configuration in my OS (Windows). Check the following links for details:
Where does the socket timeout of 21000 ms come from?
Which is the default TCP connect timeout in Windows?
Shortly, based on articles mentioned in the links above, Windows uses 3000ms initial timeout (InitialRto setting) and does 2 retries with doubled timeout from the previous attempt (MaxSynRetransmissions setting): 3sec + 2*3sec + 4*3sec = 21 sec.
In order to increase this timeout you can set more retries with the following command:
netsh interface tcp set global MaxSynRetransmissions=3
I use Spring Integration 3.0.0 with Active MQ 5.11.1 it works without any problem. but I noticed that when I stop Active MQ, i get error in my logs every 5 seconds.
if you have an idea for this problem?
Error :
ERROR [org.springframework.jms.listener.DefaultMessageListenerContainer#0-1] [DefaultMessageListenerContainer] Could not refresh JMS Connection for destination 'topic' - retrying in 5000 ms. Cause: Could not connect to broker URL: localhost. Reason: java.net.ConnectException: Connection refused: connect
When the listener container loses the connection, it tries to reconnect every 5 seconds by default until the broker is running again.
You can configure the time and/or add an exponential back off. See setRecoveryInterval and setBackOff.
Or, call stop() on the container to stop the attempts.
Call start() to start again.