We are trying to write a message to a broker queue. But the whole request fails when it tries to commit the JMS transaction & then it tries to rollback each subsequent time. We use oracle XA drivers. Not sure where to post this issue: MQ forums or Oracle forum. So thought would give a try here. Can someone help resolve this please.
Error:
[9/25/12 17:10:06:871 EDT] 0000003e XATransaction E J2CA0027E: An exception occurred while invoking commit on an XA Resource Adapter from dataSource JMS$QCF$JMSManagedConnection#23, within transaction ID {XidImpl: formatId(57415344), gtrid_length(36), bqual_length(54), data(00000139ff43ef2500000001000043106c82332ef6bc723402e84f341fb357080ddd4d1b00000139ff43ef2500000001000043106c82332ef6bc723402e84f341fb357080ddd4d1b000000010000000000000000000000000001)}: javax.transaction.xa.XAException: The method 'xa_commit' has failed with errorCode '-7'.
at com.ibm.mq.jmqi.JmqiXAResource.commit(JmqiXAResource.java:407)
at com.ibm.ejs.jms.JMSManagedSession$JMSXAResource.commit(JMSManagedSession.java:1702)
at com.ibm.ejs.j2c.XATransactionWrapper.commit(XATransactionWrapper.java:463)
at com.ibm.ws.Transaction.JTA.JTAXAResourceImpl.commit_one_phase(JTAXAResourceImpl.java:305)
at com.ibm.ws.Transaction.JTA.RegisteredResources.flowCommitOnePhase(RegisteredResources.java:2916)
at com.ibm.ws.Transaction.JTA.TransactionImpl.commitXAResources(TransactionImpl.java:2533)
at com.ibm.ws.Transaction.JTA.TransactionImpl.stage1CommitProcessing(TransactionImpl.java:1687)
at com.ibm.ws.Transaction.JTA.TransactionImpl.processCommit(TransactionImpl.java:1647)
at com.ibm.ws.Transaction.JTA.TransactionImpl.commit(TransactionImpl.java:1582)
at com.ibm.ws.Transaction.JTA.TranManagerImpl.commit(TranManagerImpl.java:247)
at com.ibm.ws.Transaction.JTA.TranManagerSet.commit(TranManagerSet.java:168)
at com.ibm.ws.Transaction.JTA.UserTransactionImpl.commit(UserTransactionImpl.java:293)
at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1009)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1002)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:901)
at org.springframework.scheduling.commonj.DelegatingWork.run(DelegatingWork.java:61)
at com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java:264)
at java.security.AccessController.doPrivileged(Native Method)
at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:1137)
at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:195)
at com.ibm.ws.asynchbeans.CJWorkItemImpl.run(CJWorkItemImpl.java:187)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1560)
.
[9/25/12 17:10:06:880 EDT] 0000003e RegisteredRes W WTRN0052E: An attempt by the transaction manager to call one phase commit on a transactional resource has resulted in an XAER_RMFAIL error. The resource was com.ibm.ws.Transaction.JTA.JTAXAResourceImpl#1d07bf1#{XidImpl: formatId(57415344), gtrid_length(36), bqual_length(54), data(00000139ff43ef2500000001000043106c82332ef6bc723402e84f341fb357080ddd4d1b00000139ff43ef2500000001000043106c82332ef6bc723402e84f341fb357080ddd4d1b000000010000000000000000000000000001)}
[9/25/12 17:10:06:887 EDT] 0000003e DefaultMessag W org.springframework.jms.listener.DefaultMessageListenerContainer handleListenerSetupFailure Setup of JMS message listener invoker failed for destination 'queue:///RANDOM QUEUE?targetClient=1' - trying to recover. Cause: Heuristic completion: outcome state is mixed; nested exception is javax.transaction.HeuristicMixedException
Here's the cause and resolution
Quote-
The cause of these errors is usually the result of a WebSphere MQ
messaging provider JMS Connection being closed off by WebSphere
Application Server because the Aged timeout for the Connection has
expired.
Resolution-
To resolve this issue, ensure that the JMS Connection Factory being
used by the application has the Connection Pool property Aged timeout
set to zero. This will prevent JMS Connections being closed when they
are returned to the Free Pool, and so ensures that any outstanding
transactional work can be completed
It is sometimes also caused by the faulty DataDirect Driver and is reported and fixed by IBM, see this.
Earlier we had multiple JMS sessions. That was probably the cause of the issue in one of the environments. So we had to change to 2 diff. sessions & now it works!
Julian:
My scenario is slightly different from yours. Earlier we had:
A request message was put on the queue. This queue was picked up & processed. Then we saved to the DB & then generated another message & put it on another broker queue & then sent a response to the first message. All this was 1 flow.
Now we changed that to 2 diff. flows : Request --> Process --> Save to DB --> Reply
and then another flow to put on Broker queue.
Hope this helps
Related
While initiating flow from spring web server by passing required values from PartyA to PartyB in corda, I am getting following exception in my initiating node PartyA, kindly do the needfull.
entered verifysend method
E 12:01:47+0530 [Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3#6cb2b947)] core.client.createConnection - AMQ214016: Failed to create netty connection
javax.net.ssl.SSLException: handshake timed out
at io.netty.handler.ssl.SslHandler.handshake(...)(Unknown Source) ~[netty-all-4.1.9.Final.jar:4.1.9.Final]
=========collecting ended=========
Even through my flow is running successfully by giving response as Transaction id:.... commited to ledger in web server, but my flow is taking around 5 minutes to create a unconsumed state.
I think you are run into low-memory issue, so one of your nodes was crashed. Hence lead to the handshake error.
The current minimal requirement for starting a node is 1GB of JVM Heap and 2GB minimal host RAM.
I have installed one J2EE application in Websphere ND 8.5.5.9 on a IBM AIX 7.2 server.
While installing application, I have skipped the Queue setup by giving the dummy values to it. Then, Listener port issue came up, as the queue was trying to connect to dummy setup. This way the connection pool was full and system started giving exceptions. So, I re-installed the application and kept the Listener port in STOP mode. First few hours application ran as expected. Now, it is giving below exceptions:
[5/23/18 17:29:53:609 CEST] 000000a9 FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/"".
[5/23/18 17:31:12:899 CEST] 00000055 FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/"".
[5/23/18 17:31:12:900 CEST] 00000055 AlarmThreadMo W UTLS0009W: Alarm Thread "Non-deferrable Alarm : 0" (00000055) previously reported to be delayed has now completed. It was active for approximately 180004 milliseconds.
[5/23/18 17:32:11:191 CEST] 00000029 AlarmThreadMo W UTLS0008W: The return of alarm thread "Non-deferrable Alarm : 2" (00000057) to the alarm thread pool has been delayed for 18271 milliseconds. This may be preventing normal alarm function within the application server. The alarm listener stack trace is as follows:
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:201)
at com.ibm.ejs.j2c.FreePool.queueRequest(FreePool.java:438)
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1344)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3898)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3118)
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1548)
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:1031)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:646)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:924)
at com.ibm.ws.extensionhelper.db.impl.DatabaseHelperImpl$DSWrapper.getConnection(DatabaseHelperImpl.java:1595)
at com.ibm.ws.extensionhelper.db.impl.DatabaseHelperImpl.getConnection(DatabaseHelperImpl.java:750)
at com.ibm.ws.leasemanager.impl.LeaseManagerDBHelper.getConnection(LeaseManagerDBHelper.java:213)
at com.ibm.ws.leasemanager.impl.LeaseStoreImpl.renew(LeaseStoreImpl.java:452)
at com.ibm.ws.leasemanager.impl.LeaseImpl.renew(LeaseImpl.java:141)
at com.ibm.ws.scheduler.LeaseAlarm.alarm(LeaseAlarm.java:173)
at com.ibm.ejs.util.am._Alarm.runImpl(_Alarm.java:151)
at com.ibm.ejs.util.am._Alarm.run(_Alarm.java:136)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892).
Please suggest what can be done to free the connection pool without making any code changes. Is it possible to handle it on OS level or Websphere level?
The last of the warnings with the 18 second wait is for a connection attempt that is made by the WAS scheduler. You should look in your configuration to see if the scheduler is configured to use the same data source, jdbc/"" (which is an unusual name - is this data source configured properly?) as the prior errors. There are a couple of possibilities for the cause behind theses errors/warnings. You could have a connection pool that is insufficiently sized to handle the load that your application requires, or you could have code that is holding onto connections for too long, starving out the other users of the data source.
We have a web application with the stack Spring, Hibernate, C3P0, Oracle DB Driver (habing an Oracle DB behind).
From time to time we experience blocking locks over a longer period of time which then get killed on the DB end. (we know this is caused by bad application design and we will fix it, but it's not the point of this quesion).
After the DB session was killed by DB it seems that the connection pool reuses the now broken connection which results in the error:
A PooledConnection that has already signalled a Connection error is still in use!
Another error has occurred [ java.sql.SQLRecoverableException: Closed Connection ] which will not be reported to listeners!
On the DataSource we configured
dataSource.setTestConnectionOnCheckin(true);
dataSource.setTestConnectionOnCheckout(true);
But it did not help. We expected that the connections fail these tests and then get renewed. But this does not happen.
Any hints for us how to recreate the broken connections?
This warning is given when a Connection that is already checked out experiences an Exception that causes c3p0 to treat it as invalid (so it will not be reincorporated back into the pool on close()), but the Connection continues to be used and experiences an Exception again. These are not broken Connections in the pool. They are broken Connections in-use by the application. So testing them on checkout (or checkin) doesn't do anything about them.
To get rid of this, you need to examine the Exception handling within your application code. Are there circumstances where an invalid Connection might have thrown an Exception, but that Exception was caught and the Connection reused?
The warning itself is harmless. It's just saying c3p0 already knows the Connection is bad, it won't emit an event to signal that again.
I am using IBM Websphere MQ 7.5 in Unix system. I have installed the client on my machine and server is running on other machine. I am observing a scenario where I am able to communicate with server when running my JMS application via 'mqm' user but facing below mentioned error when using other user.
But I am able to run 'amqsputc' and 'amqsgetc' command and communicate with the server with mqm as well as other user also. I have followed all steps mentioned http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.ins.doc/q009300_.htm?lang=en
Exception:
com.ibm.msg.client.jms.DetailedJMSSecurityException: JMSWMQ2013: The security authentication was not valid that was supplied for QueueManager 'TestManager' with connection mode 'Client' and host name 'x.x.x.x(9923)'.
Please check if the supplied username and password are correct on the QueueManager to which you are connecting.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:521)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:221)
at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:426)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7ProviderConnection(WMQConnectionFactory.java:6902)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:6277)
at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:285)
at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6233)
at com.ibm.mq.jms.MQQueueConnectionFactory.createQueueConnection(MQQueueConnectionFactory.java:120)
at com.ibm.mq.jms.MQQueueConnectionFactory.createConnection(MQQueueConnectionFactory.java:203)
at performance.IBMMQTestProducer.start(IBMMQTestProducer.java:142)
at performance.IBMMQTestProducer.main(IBMMQTestProducer.java:177)
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2035' ('MQRC_NOT_AUTHORIZED').**
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:209)
... 9 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2035;AMQ9509: Program cannot open queue manager object. [1=2035,5=???]**
at com.ibm.mq.jmqi.internal.JmqiTools.getQueueManagerInfo(JmqiTools.java:783)
at com.ibm.mq.jmqi.remote.impl.RemoteSession.loadInfo(RemoteSession.java:1993)
at com.ibm.mq.jmqi.remote.impl.RemoteSession.getName(RemoteSession.java:2026)
at com.ibm.mq.jmqi.remote.api.RemoteHconn.getName(RemoteHconn.java:728)
at com.ibm.mq.ese.intercept.JmqiConnInterceptorImpl.validate(JmqiConnInterceptorImpl.java:321)
at com.ibm.mq.ese.intercept.JmqiConnInterceptorImpl.afterConnect(JmqiConnInterceptorImpl.java:226)
at com.ibm.mq.ese.intercept.JmqiConnInterceptorImpl.afterJmqiConnect(JmqiConnInterceptorImpl.java:133)
at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect(InterceptedJmqiImpl.java:315)
at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect(ESEJMQI.java:337)
I am able to run it when I am running as myself and passing 'mqm' when starting the connection.
connection = cf.createConnection("mqm", "pswd");
I am not getting anything in manager's log. Below is the log.
------------------------------------------------------------------------------
10/30/2015 06:50:54 AM - Process(31064.1) User(mqm) Program(strmqm)
Host(x.x.x.x) Installation(Installation1)
VRMF(7.5.0.2)
AMQ7125: There are 83 days left in the trial period for this copy of WebSphere
MQ.
EXPLANATION:
This copy of WebSphere MQ is licensed for a limited period only.
ACTION:
None.
Given that you have told us that you have successfully connected and run the amqsgetc and amqsputc client samples using the same server-connection channel as you are attempting to use for your JMS program that suggests that this is not a connection time problem, in other words the MQCONN to the queue manager has been successful and something following that is failing. We know that you can MQPUT and MQGET (since that is what the aforementioned samples do).
Something that JMS does that those simple samples do not do, is an MQINQ of the queue manager. The following part of your exception makes me wonder if that is what you are tripping over:-
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2035;
AMQ9509: Program cannot open queue manager object. [1=2035,5=???] at
com.ibm.mq.jmqi.internal.JmqiTools.getQueueManagerInfo(JmqiTools.java:783) at
To be completely certain you must check the queue manager AMQERR01.LOG to see what is reported there. If it is missing authorization then it will tell you there.
When standalone client and jboss server runs on the same machine, everything works fine, but when i try to run the JBoss on a VM, i can get a JNDIContext from the Server,i can do a successful lookup, but i cannot establish a connection.
The Client throwing an exception as:
Exception in thread "main" javax.jms.JMSException: Failed to create session factory
at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:605)
at #org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:119)
...
Caused by: HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]
at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:769)
at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:601)
... 2 more
And the server End showing msg as,
JBAS011806: Channel end notification received, closing channel Channel ID 459d0f32 (inbound) of Remoting connection 00e9212a to /192.168.35.29:42179
I have seen other posts also related to this issue , but I didnt clear with the solution . So Plz help me out and forgive me If I am asking silly question since I am new in this domain.
Thanks.
You have to configure the port on the server. You probably have the server starting as localhost and you are connecting externally, in such way the JNDI can connect but not the HornetQ connector.
You certainly have a configuration error, but without knowing version, what actual message you got and configuration options I have little information to help you more beyond that.