I see occasional stack traces like the following when the Spring Boot 2.0 server handles an incoming request. I am using Spring Boot 2.2.4 with spring-boot-starter-jetty. This exception only happens around 0.1% of time. Has anyone seen something similar and has any idea on why this would happen? The service is running in AWS EKS behind an Application Load Balancer.
j.u.c.TimeoutException: Idle timeout expired: 30001/30000 ms
at o.e.j.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)
at o.e.j.io.IdleTimeout.idleCheck(IdleTimeout.java:113)
at j.u.c.Executors$RunnableAdapter.call(Executors.java:515)
at j.u.c.FutureTask.run(FutureTask.java:264)
at j.u.c.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at j.u.c.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at j.u.c.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 common frames omitted
Suppressed: java.lang.Throwable: HttpInput failure
at o.e.j.s.HttpInput.failed(HttpInput.java:830)
at o.e.j.s.HttpConnection$BlockingReadCallback.failed(HttpConnection.java:656)
at o.e.j.i.FillInterest.onFail(FillInterest.java:138)
at o.e.j.i.AbstractEndPoint.onIdleExpired(AbstractEndPoint.java:407)
... 8 common frames omitted
Wrapped by: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
at o.e.j.s.HttpInput$ErrorState.noContent(HttpInput.java:1083)
at o.e.j.s.HttpInput.read(HttpInput.java:321)
at c.r.p.e.s.c.s.MeasurableRequestWrapper$CountingInputStream.read(MeasurableRequestWrapper.java:62)
at j.i.FilterInputStream.read(FilterInputStream.java:133)
at j.i.PushbackInputStream.read(PushbackInputStream.java:183)
at c.f.j.c.j.UTF8StreamJsonParser._loadMore(UTF8StreamJsonParser.java:220)
at c.f.j.c.j.UTF8StreamJsonParser.parseEscapedName(UTF8StreamJsonParser.java:1985)
at c.f.j.c.j.UTF8StreamJsonParser.parseLongName(UTF8StreamJsonParser.java:1872)
at c.f.j.c.j.UTF8StreamJsonParser.parseMediumName2(UTF8StreamJsonParser.java:1810)
at c.f.j.c.j.UTF8StreamJsonParser.parseMediumName(UTF8StreamJsonParser.java:1767)
at c.f.j.c.j.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1702)
at c.f.j.c.j.UTF8StreamJsonParser.nextFieldName(UTF8StreamJsonParser.java:1029)
at c.f.j.d.d.s.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:512)
at c.f.j.d.d.s.MapDeserializer.deserialize(MapDeserializer.java:364)
at c.f.j.d.d.s.MapDeserializer.deserialize(MapDeserializer.java:29)
at c.f.j.d.d.s.CollectionDeserializer.deserialize(CollectionDeserializer.java:286)
... 124 common frames omitted
Wrapped by: c.f.j.d.JsonMappingException: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms (through reference chain: com.abc.service.api.resources.SalesDataRequest["details"]->java.util.ArrayList[0]->com.abc.service.api.resources.SalesDataRequest$SalesDataDetail["events"]->java.util.ArrayList[2])
at c.f.j.d.JsonMappingException.wrapWithPath(JsonMappingException.java:394)
at c.f.j.d.JsonMappingException.wrapWithPath(JsonMappingException.java:365)
at c.f.j.d.d.s.CollectionDeserializer.deserialize(CollectionDeserializer...
You had a connection that was supplying data on the request body (hence the HttpInput references).
The data was in a blocking read (HttpInput.read()) waiting for more data.
Then the idle time out kicked in for that connection and the read was failed with HttpInput.noContent() indicating that the read was incomplete, with a nested cause of Idle Timeout.
This is a standard idle timeout scenario.
Not that unusual or remarkable.
Quite common with mobile clients (mobile tower switch, poor network connectivity, switching from cell tower to wifi, or back again, etc). It even happens with browsers on laptops (wifi issues, browser crashes, battery issues, lid closed, sleep mode, etc..)
The fact that it happens with only 1% of your requests is actually quite remarkable (usually the percentage of requests with network issues is higher on a public facing web site)
Related
We suddenly started receiving following error in out application services, we did not make any change to config in recent time. Could any one provide some information regarding error?
Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: xxx service could not be queued for execution and no fallback available.
Caused by: java.util.concurrent.RejectedExecutionException: Rejected command because thread-pool queueSize is at rejection threshold.
Configuration in gateway are:
hystrix.command.default.execution.isolation.strategy=THREAD
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds=360000
hystrix.threadpool.default.coreSize=40
hystrix.command.default.circuitBreaker.forceClosed=true
hystrix.threadpool.default.maxQueueSize=2000
hystrix.threadpool.default.queueSizeRejectionThreshold=1800
zuul.routes.<instance>.path=/<instance>/**
zuul.routes.<instance>.serviceId=<instance>
zuul.routes.<instance>.sensitiveHeaders=
<instance>.ribbon.OkToRetryOnAllOperations=true
<instance>.ribbon.MaxAutoRetriesNextServer=1
<instance>.ribbon.MaxAutoRetries=0
hystrix.command.<instance>.execution.isolation.strategy=THREAD
hystrix.command.<instance>.execution.isolation.thread.timeoutInMilliseconds=360000
hystrix.threadpool.<instance>.coreSize=40
hystrix.command.<instance>.circuitBreaker.forceClosed=true
hystrix.threadpool.<instance>.maximumSize=100
hystrix.threadpool.<instance>.queueSizeRejectionThreshold=80
We were having the coreSize, forceClosed, maxQueueSize and quoteSizeRejectionThreshold as default and not it is configured this level. It still has the issue. We tried restart Gateway and Eureka instance.
While initiating flow from spring web server by passing required values from PartyA to PartyB in corda, I am getting following exception in my initiating node PartyA, kindly do the needfull.
entered verifysend method
E 12:01:47+0530 [Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3#6cb2b947)] core.client.createConnection - AMQ214016: Failed to create netty connection
javax.net.ssl.SSLException: handshake timed out
at io.netty.handler.ssl.SslHandler.handshake(...)(Unknown Source) ~[netty-all-4.1.9.Final.jar:4.1.9.Final]
=========collecting ended=========
Even through my flow is running successfully by giving response as Transaction id:.... commited to ledger in web server, but my flow is taking around 5 minutes to create a unconsumed state.
I think you are run into low-memory issue, so one of your nodes was crashed. Hence lead to the handshake error.
The current minimal requirement for starting a node is 1GB of JVM Heap and 2GB minimal host RAM.
I have installed one J2EE application in Websphere ND 8.5.5.9 on a IBM AIX 7.2 server.
While installing application, I have skipped the Queue setup by giving the dummy values to it. Then, Listener port issue came up, as the queue was trying to connect to dummy setup. This way the connection pool was full and system started giving exceptions. So, I re-installed the application and kept the Listener port in STOP mode. First few hours application ran as expected. Now, it is giving below exceptions:
[5/23/18 17:29:53:609 CEST] 000000a9 FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/"".
[5/23/18 17:31:12:899 CEST] 00000055 FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/"".
[5/23/18 17:31:12:900 CEST] 00000055 AlarmThreadMo W UTLS0009W: Alarm Thread "Non-deferrable Alarm : 0" (00000055) previously reported to be delayed has now completed. It was active for approximately 180004 milliseconds.
[5/23/18 17:32:11:191 CEST] 00000029 AlarmThreadMo W UTLS0008W: The return of alarm thread "Non-deferrable Alarm : 2" (00000057) to the alarm thread pool has been delayed for 18271 milliseconds. This may be preventing normal alarm function within the application server. The alarm listener stack trace is as follows:
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:201)
at com.ibm.ejs.j2c.FreePool.queueRequest(FreePool.java:438)
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1344)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3898)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3118)
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1548)
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:1031)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:646)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:924)
at com.ibm.ws.extensionhelper.db.impl.DatabaseHelperImpl$DSWrapper.getConnection(DatabaseHelperImpl.java:1595)
at com.ibm.ws.extensionhelper.db.impl.DatabaseHelperImpl.getConnection(DatabaseHelperImpl.java:750)
at com.ibm.ws.leasemanager.impl.LeaseManagerDBHelper.getConnection(LeaseManagerDBHelper.java:213)
at com.ibm.ws.leasemanager.impl.LeaseStoreImpl.renew(LeaseStoreImpl.java:452)
at com.ibm.ws.leasemanager.impl.LeaseImpl.renew(LeaseImpl.java:141)
at com.ibm.ws.scheduler.LeaseAlarm.alarm(LeaseAlarm.java:173)
at com.ibm.ejs.util.am._Alarm.runImpl(_Alarm.java:151)
at com.ibm.ejs.util.am._Alarm.run(_Alarm.java:136)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892).
Please suggest what can be done to free the connection pool without making any code changes. Is it possible to handle it on OS level or Websphere level?
The last of the warnings with the 18 second wait is for a connection attempt that is made by the WAS scheduler. You should look in your configuration to see if the scheduler is configured to use the same data source, jdbc/"" (which is an unusual name - is this data source configured properly?) as the prior errors. There are a couple of possibilities for the cause behind theses errors/warnings. You could have a connection pool that is insufficiently sized to handle the load that your application requires, or you could have code that is holding onto connections for too long, starving out the other users of the data source.
I am getting socket exception for broken pipe in my client side.
[write] I/O error: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Broken pipe (Write failed)
[LoggingManagedHttpClientConnection::shutdown] http-outgoing-278: Shutdown connection
1520546494584[20180308 23:01:34] [ConnectionHolder::abortConnection] Connection discarded
1520546494584[20180308 23:01:34] [BasicHttpClientConnectionManager::releaseConnection] Releasing connection [Not bound]
It seems that the upgradation of httpclient jar is causing issue.
Issue is not coming with httpclient-4.3.2
Exception is coming in every 2 minutes. Issue is intermittent at times.
after , send expect:100-continue ,conn.flush is throwing exception
client and server are Linux machine
client uses http jar to make request to server REST.
Please help me in debugging the issue
can httpjar cause such issue?
The persistent connections that are kept alive by the connection manager become stale. That is, the target server shuts down the connection on its end without HttpClient being able to react to that event, while the connection is being idle, thus rendering the connection half-closed or 'stale'
This is a general limitation of the blocking I/O in Java. There is simply no way of finding out whether or not the opposite endpoint has closed connection other than by attempting to read from the socket.
If a stale connection is used to transmit a request message the request execution usually fails in the write operation with SocketException and gets automatically retried.
Apache HttpClient works this problem around by employing the so stale connection check which is essentially a very brief read operation. However, the check can and often is disabled. In fact it is often advisable to have it disabled due to extra latency the check introduces.
The handling of stale connections was changed in version 4.4. Previously, the code would check every connection by default before re-using it. The code now only checks the connection if the elapsed time since the last use of the connection exceeds the timeout that has been set. The default timeout is set to 2000ms
I have increased the concurrent tasks to be '10' for PutSQL processor.
At that time it shows below error but there is no data loss.
failed to process due to org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object; rolling back session:
if i have remove concurrent tasks then it worked without those exception
while google this exception i have found answer in below link
I am getting Cannot get a connection, pool error Timeout waiting for idle object, When I try to create more than 250 threads in my web application
But i don't know how to avoid this issue in NiFi putSQL.
Can anyone help me to resolve this?
This exception occurs when the pool manager cannot produce a viable connection to a waiting requester and the maxWait has passed therefore triggering a timeout.
There are several causes, but they usually fall into 2 main categories:
The DB is down or unreachable.
The connection pool (which is set to 100 max active) is out of connections.
DBCPConnectionPool controller service in nifi has 8 max connection by default and 500 milli seconds max wait time. When PutSQL processor occupied 8 connection from DBCP pool and when it request for 9th connection or exceed the max connection limit then it will throw "Cannot get a connection" exception.
You can try 2 things to avoid this Exception :
You can increase the "Max Wait Time" in DBCPConnectionPool controller
service configuration.
You can increase the "Max Total Connections" limit in
DBCPConnectionPool controller service configuration.
Kindly find the below screenshot where you need to do changes.
It might resolve your issue.
This exception can occurs if some connections are never closed so they do not become available in the pool again.
So more and more connections remain open until reaching the max.
Make sure all threads are closing the connections used.