I have a setup with a MQ Queue and a Liberty application taking messages from MQ an puts them into a database.
After random number of messages going nice and clean through I get a timeout exception (see below) When this happens it seems like the thread hangs utilizing 100% CPU, the more this happens the higher load on the server. Any suggestions?
com.ibm.tx.jta.embeddable.impl.EmbeddableTimeoutManager I WTRN0124I: When the timeout occurred the thread with which the transaction is, or was most recently, associated was Thread[Default Executor-thread-37,5,Default Executor Thread Group]. The stack trace of this thread when the timeout occurred was: com.ibm.mq.jmqi.remote.impl.RemoteSession.receiveOneTSH(RemoteSession.java:884) com.ibm.mq.jmqi.remote.impl.RemoteSession.receive1stGetReplyTSH(RemoteSession.java:1509) com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiGetMessageWithProps(RemoteFAP.java:9820) com.ibm.mq.jmqi.remote.api.RemoteFAP.MQGET(RemoteFAP.java:9511) com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.MQGET(InterceptedJmqiImpl.java:1213) com.ibm.mq.ese.jmqi.ESEJMQI.MQGET(ESEJMQI.java:430) com.ibm.msg.client.wmq.compat.base.internal.MQSESSION.MQGET(MQSESSION.java:975) com.ibm.msg.client.wmq.compat.base.internal.MQQueue.getMsg2Int(MQQueue.java:1355) com.ibm.msg.client.wmq.compat.base.internal.MQQueue.getMsg2NoExc(MQQueue.java:1445) com.ibm.msg.client.wmq.compat.jms.internal.MQSession.consume(MQSession.java:1865) com.ibm.msg.client.wmq.compat.jms.internal.MQSession.loadMessageReference(MQSession.java:4224) com.ibm.msg.client.jms.internal.JmsSessionImpl.consume(JmsSessionImpl.java:3639) com.ibm.msg.client.jms.internal.JmsSessionImpl.run(JmsSessionImpl.java:3227) com.ibm.mq.jms.MQSession.run(MQSession.java:937) com.ibm.mq.connector.inbound.ASFWorkImpl.doDelivery(ASFWorkImpl.java:110) com.ibm.mq.connector.inbound.AbstractWorkImpl.run(AbstractWorkImpl.java:229) com.ibm.ws.jca.inbound.security.JCASecurityContextService.runInInboundSecurityContext(JCASecurityContextService.java:49) com.ibm.ws.jca.internal.WorkProxy.run(WorkProxy.java:354) com.ibm.ws.context.service.serializable.ContextualRunnable.run(ContextualRunnable.java:79) com.ibm.ws.jca.internal.WorkProxy.call(WorkProxy.java:285) com.ibm.ws.jca.internal.WorkProxy.call(WorkProxy.java:58) java.util.concurrent.FutureTask.run(FutureTask.java:277) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) java.lang.Thread.run(Thread.java:811)
Related
We are getting the below error while executing the search requests. We are using ElasticSearch 7.9.2v
java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED
I am only using sync method to execute the requests as shown below:
client.search(searchRequest,RequestOptions.DEFAULT)
When the first time the issue occurs looks like it stops the RestHighLevelClient and the subsequent calls are getting failed with the same error. we need to restart our app to initialize the client again. From last two days, we are running into this issue very frequently.
Note: we are not closing the client for every call. We initiate the client during application startup and close it only when the app is getting shut down.
Restarting the java process (in my case is SpringBoot java process) fixes the problem.
Had the same issue. It was caused by OutOfMemoryError that was caused by other changes at my service.
So increasing heap-space limit helped.
Elasticsearch.Net.UnexpectedElasticsearchClientException: There were not enough free threads in the ThreadPool to complete the operation. ---> System.InvalidOperationException: There were not enough free threads in the ThreadPool to complete the operation
If i use the Search method of the IElasticClient iterface the operation performs the search successfully.
Also .. i have a hard time believing it's a setting on the elastic server since the call works fine on another machine.
Any ideas what thread pool this is referring to? Much appreciated.
Discovered there was a piece of code in the application that was setting the threadpool
ThreadPool.SetMaxThreads(maxWorkerCount, maxIOCount);
because the virtual server was set to have only 2 processors.. so this code set the max threads to 2. This caused the async calls of the application on that machine to throw those errors.
I am getting the stuck thread error often while trying to send the JMS message to another manager server within the domain in our production environment.
Initially, we felt that it might be due to Load on the server but issue occurred randomly Even at the time of less load and system processing well some high volume time
We are not able to find the reason for the same.
Error Information:
weblogic.jms.client.JMSConnectionFactory.createQueueConnection(JMSConnectionFact
ory.java:199)
The term stuck thread is commonly used in Oracle WebLogic Server.
What is a stuck thread and why is thread diagnosed as a stuck thread?
When does stuck thread occur?
What kind of impact does it happen to
the running application?
What are the prevention mechanisms?
Stuck Threads are threads that are blocked, and can't return to the threadpool for a certain amout of time. By Default, the WLS comes with 600 secs. If some thread doesn't return in 600 secs, It gets a flag 'stuck thread'.
– > Stuck Threads are only flags, there to warn you that this thread is taking too long.
Searching a bit on google I found this site: http://www.munzandmore.com/2012/ora/weblogic-stuck-threads-howto
It explains what are stuck threads, as well some methods to work around them.
While tring to start websphere server it is throwing the error:
thread server startup:2'' (0000000a) has been active for 786590 milliseconds and may be hung.
How to resolve this WSVR0605W error?
That error will always be displayed when a thread has been running for a certain time. It is not for certain that this actually indicates an error, it depends on what your applications do on your server. If you have an operation that runs within the same thread for more than 10 minutes, this error will eventually pop up even though it is within the normal operation of your application. If this is the case you will later get a message of the style [10/29/13 8:23:11:008 CET] 00004709 ThreadMonitor W WSVR0606W: Thread "Thread_ID" (00004709) was previously reported to be hung but has completed. It was active for approximately 727968 milliseconds. There is/are 0 thread(s) in total in the server that still may be hung.
If you do not have anything that should be running for that long, the thread may indeed be hung and you need to investigate what operation exactly is hung. Start by examining the logs using the same thread ID as the one reported as possibly hung.