I am getting hung thread in WAS - websphere

Question

[5/10/16 14:57:10:494 WIT] 00000003 ThreadMonitor W WSVR0605W: Thread "WebContainer : 11" (000000ec) has been active for 749826 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:140)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at com.manulife.mail.archive.GenerateReport.generate(GenerateReport.java:88)
at sun.reflect.GeneratedMethodAccessor841.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at teamworks.connector.JavaReflectionConnector$1.execute(JavaReflectionConnector.java:81)
at com.lombardisoftware.server.core.ManagedAssetClassLoaderUtil.executeWithManagedAssetClassLoader(ManagedAssetClassLoaderUtil.java:53)
at teamworks.connector.JavaReflectionConnector.execute(JavaReflectionConnector.java:49)
at com.lombardisoftware.component.javaconnector.worker.JavaConnectorWorker.invokeJavaMethod(JavaConnectorWorker.java:149)
at com.lombardisoftware.component.javaconnector.worker.JavaConnectorWorker.doJob(JavaConnectorWorker.java:82)
at com.lombardisoftware.component.common.workflow.ExecutionJob.doJob(ExecutionJob.java:403)
at com.lombardisoftware.server.ejb.workflow.EJBWorkflowManagerBean.doResumeWorkflowEngine(EJBWorkflowManagerBean.java:1061)
at com.lombardisoftware.server.ejb.workflow.EJBWorkflowManagerBean.resumeTask(EJBWorkflowManagerBean.java:303)
at com.lombardisoftware.server.ejb.workflow.EJBWorkflowManagerBean.resumeTask(EJBWorkflowManagerBean.java:267)
at com.lombardisoftware.server.ejb.workflow.EJSRemoteStatefulEJBWorkflowManager_82478d70.resumeTask(Unknown Source)
at com.lombardisoftware.server.ejb.workflow._EJBWorkflowManagerInterface_Stub.resumeTask(_EJBWorkflowManagerInterface_Stub.java:287)
at com.ibm.bpm.rest.impl.service.ServiceRunner$TaskRunner.runService(ServiceRunner.java:696)
at com.ibm.bpm.rest.impl.task.StartActionHandler.handleAction(StartActionHandler.java:126)
at com.ibm.bpm.rest.impl.task.TaskResource.processTaskAction(TaskResource.java:383)
at com.ibm.bpm.rest.impl.task.TaskResource.processPostTaskAction(TaskResource.java:504)
at sun.reflect.GeneratedMethodAccessor170.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.apache.wink.server.internal.handlers.InvokeMethodHandler.handleRequest(InvokeMethodHandler.java:51)
at org.apache.wink.server.handlers.AbstractHandler.handleRequest(AbstractHandler.java:33)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:61)
at org.apache.wink.server.internal.handlers.CreateInvocationParametersHandler.handleRequest(CreateInvocationParametersHandler.java:48)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:61)
at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleSubResourceMethod(FindResourceMethodHandler.java:178)
at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleRequest(FindResourceMethodHandler.java:105)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:61)
at org.apache.wink.server.internal.handlers.FindRootResourceHandler.handleRequest(FindRootResourceHandler.java:93)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:61)
at org.apache.wink.server.internal.handlers.HeadMethodHandler.handleRequest(HeadMethodHandler.java:53)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:61)
at org.apache.wink.server.internal.handlers.OptionsMethodHandler.handleRequest(OptionsMethodHandler.java:46)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:61)
at org.apache.wink.server.internal.handlers.SearchResultHandler.handleRequest(SearchResultHandler.java:33)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:61)
at org.apache.wink.server.handlers.AbstractHandlersChain.run(AbstractHandlersChain.java:48)
at org.apache.wink.server.internal.RequestProcessor.handleRequestWithoutFaultBarrier(RequestProcessor.java:157)
at org.apache.wink.server.internal.RequestProcessor.handleRequest(RequestProcessor.java:133)
at org.apache.wink.server.internal.servlet.RestServlet.service(RestServlet.java:109)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.websphere.jaxrs.server.IBMRestServlet.service(IBMRestServlet.java:45)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1658)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1598)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:149)
at com.ibm.bpm.rest.filter.BPMRestServletFilter.doFilter(BPMRestServletFilter.java:130)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:125)
at com.ibm.bpm.servlet.filters.CompressionFilter.doFilter(CompressionFilter.java:120)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:125)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:80)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:935)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:503)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3954)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:276)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:942)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:277)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.j1646)

Answer 1

Looks like the application is making an http request to another server and it is waiting for the response to complete.
Most likely the response is either incomplete or there were network issues with the connection.
Your best bet is to place a timeout on the connection so your application thread doesn't hang out for say more than 30 secs waiting for a response. I'm not sure if the http client implementation you are using can set a timeout. If not you should be able to change implementations.
As to figure out why the response is failing to complete you can use tcpdump or iptrace to collect a network level packet trace and use Wireshark to look at the data flowing between the two machines. I would capture data both on the WAS app and on the remote server as you may see issues only on one side if it is a bad switch or retransmission issue.