Granting Oracle permissions doesn't fix WebSphere XAER_RMERR exception? - oracle

I have an application deployed on WebSphere ND 7.0.0.21. Within that I have an Oracle XA datasource configured. Recently the application crashed and when I restarted it, I started getting the following exception in SystemOut.log every second.
[11/6/12 13:55:38:650 GMT] 0000000c InternalGener I DSRA8205I: JDBC driver name : Oracle JDBC driver
[11/6/12 13:55:38:651 GMT] 0000000c InternalGener I DSRA8206I: JDBC driver version : 11.2.0.2.0
[11/6/12 13:55:38:661 GMT] 0000000c InternalOracl I DSRA8212I: DataStoreHelper name is: com.ibm.websphere.rsadapter.Oracle11gDataStoreHelper.
[11/6/12 13:55:38:662 GMT] 0000000c WSRdbDataSour I DSRA8208I: JDBC driver type : ""
[11/6/12 13:55:38:685 GMT] 0000000c WSRdbXaResour E DSRA0304E: XAException occurred. XAException contents and details are: The cause is : null.
[11/6/12 13:55:38:837 GMT] 0000000c WSRdbXaResour E DSRA0302E: XAException occurred. Error code is: XAER_RMERR (-3). Exception is: <null>
[11/6/12 13:55:38:837 GMT] 0000000c XARminst E WTRN0037W: The transaction service encountered an error on an xa_recover operation. The resource was com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl#7b867b86. The error code was XAER_RMERR. The exception stack trace follows: javax.transaction.xa.XAException
at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:709)
at com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.recover(WSRdbXaResourceImpl.java:1088)
at com.ibm.ws.Transaction.JTA.XARminst.recover(XARminst.java:141)
at com.ibm.ws.Transaction.JTA.XARecoveryData.recover(XARecoveryData.java:626)
at com.ibm.tx.jta.PartnerLogTable.recover(PartnerLogTable.java:389)
at com.ibm.tx.jta.RecoveryManager.resync(RecoveryManager.java:1530)
at com.ibm.tx.jta.RecoveryManager.performResync(RecoveryManager.java:2265)
at com.ibm.ws.tx.jta.RecoveryManager.performResync(RecoveryManager.java:114)
at com.ibm.tx.jta.RecoveryManager.run(RecoveryManager.java:2218)
at java.lang.Thread.run(Thread.java:736)
This seems to be a standard problem when appropriate permissions have not been granted on the database, to allow the db user access to the transaction requiring recovery - http://www-01.ibm.com/support/docview.wss?uid=swg21196663
However, I then ran the SQL on that page and restarted the application server, but the problem persisted. The issue only went away when I removed the transaction logs (described as a workaround on the IBM page).
Why would granting the permissions not fix the problem? Can something else cause the XAER_RMERR problem?

I would suggest to enable the traces as described in the relevant MustGather (http://www-01.ibm.com/support/docview.wss?uid=swg21153216) in order to see the actual error returned by the database.

removing transaction logs and partnerlogs solves the issue. of course one must be cautious to do it in a production environment with complex transaction schemes, yet without that WebSphere is unable to recover them itself.

Related

WebSphere 8.5.5 and JMS annotations crash server application

I'm deploying an app that makes use of Spring 4.2.5, Hibernate 4.2.8, and JMS 1.1 on to WebSphere 8.5.5 and Oracle 12.
Resources such as the database connection manager, and JMS connection factory are set in the server and wired into the Spring app using JNDI.
When the app starts I see this in the logs:
[3/18/16 15:18:32:717 EST] 0000008b SystemOut O [B#631dd237/Set;
at org.springframework.jms.annotation.JmsListenerAnnotationBeanPostProcessor$1.inspect(JmsListenerAnnotationBeanPostProcessor.java:202)
at org.springframework.jms.annotation.JmsListenerAnnotationBeanPostProcessor$1.inspect(JmsListenerAnnotationBeanPostProcessor.java:198)
at org.springframework.core.MethodIntrospector$1.doWith(MethodIntrospector.java:72)
at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:495)
at org.springframework.core.MethodIntrospector.selectMethods(MethodIntrospector.java:68)
at org.springframework.jms.annotation.JmsListenerAnnotationBeanPostProcessor.postProcessAfterInitialization(JmsListenerAnnotationBeanPostProcessor.java:197)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:421)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1559)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)
... 58 more
[3/18/16 15:18:32:718 EST] 0000008b SystemOut O [FIAT-CSP-NA] [WebContainer : 0] 2016-03-18 15:18:32,718 [INFO ] org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean: Closing JPA EntityManagerFactory for persistence unit default
As you can see it appears there is some sort of error in the JMSListenerAnnotationBeanPostProcessor, followed by a message indicating that the JPA entity manager is shutting down.
I presume from this that there is a JMS problem which is shutting down the app.
Has anyone seen this? Do you know what might be the issue?
I'm really stuck on this.
The answer turned out to be class path issues. (Isn't everything on Websphere!)
I needed to remove several jars from the war file that where conflicting with jars web sphere supplied.

Same query on same database gives different results on OAS 10.1.3

I'm seeing something odd when I run a query in an application deployed in Oracle Application Server 10.1.3, with Oracle10g.
When I run a statement against the database directly (e.g. a standalone app that calls a DAO implemented with hibernate) I see the following:
select
documentco0_.CONTENT_ID as CONTENT1_63_0_,
documentco0_.TSTAMP as TSTAMP63_0_,
documentco0_.CONTENT as CONTENT63_0_
from
MySchema.MyTable documentco0_
where
documentco0_.CONTENT_ID=?
[main] TRACE org.hibernate.type.LongType - binding '1768334' to parameter: 1
[main] TRACE org.hibernate.type.TimestampType - returning '2013-08-05 17:31:32' as column: TSTAMP63_0_
[main] TRACE org.hibernate.type.BinaryType - returning '7f587f608090cac6c9c68081818180b380b380807f5b80c3807f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f7f40808b8880918091818191807f44809f8080818581818181818180808080808080808182838485868788898a8b7f44803590808281838382848385858484808081fd8182838084918592a1b1c18693d1e187a2f194b201112188a3c2314195d25170a4b3e2f202898a969798999aa5a6a7a8a9aab4b5b6b7b8b9bac3c4c5c6c7c8c9cad3d4d5d6d7d8d9dae3e4e5e6e7e8e9eaf3f4f5f6f7f8f9fa030405060708090a12131415161718191a22232425262728292a32333435363738393a42434445464748494a52535455565758595a6162636465666768696a7172737475767778797a7f5a808881818080bf80fef947bf520c730eff25ada7bd007c7f807a460efd87677f805625220aab7f59' as column: CONTENT63_0_
The same DAO operation when run within the application server however returns the following:
select
documentco0_.CONTENT_ID as CONTENT1_63_0_,
documentco0_.TSTAMP as TSTAMP63_0_,
documentco0_.CONTENT as CONTENT63_0_
from
MySchema.MyTable documentco0_
where
documentco0_.CONTENT_ID=?
2013-08-06 12:49:46,484 TRACE [AJPRequestHandler-RMICallHandler-12] myuser:4 (NullableType.java:133 nullSafeSet()) - binding '1768334' to parameter: 1
2013-08-06 12:49:46,500 TRACE [AJPRequestHandler-RMICallHandler-12] myuser:4 (NullableType.java:172 nullSafeGet()) - returning '2013-08-05 17:31:32' as column: TSTAMP63_0_
2013-08-06 12:49:46,500 TRACE [AJPRequestHandler-RMICallHandler-12] myuser:4 (NullableType.java:172 nullSafeGet()) - returning '80d48081818c808080818080808180808099ff0c809a5c9d809a5c9c80828082808080817f587f608090cac6c9c68081808080804818f7ef8081808080808080808080808080808080808080809a5c9c83408c508081' as column: CONTENT63_0_
You can see that the identifier and timestamp are the same in both cases, but the content blob is different: 360 bytes in the first case and 86 bytes in the second case.
The stand-alone application uses a BasicDataSource, while the application on the server uses a JNDI data source. I have verified that the BasicDataSource contains the same JDBC url that is used in the JNDI data source. Both data sources use the same credentials.
The database operation in the application server has a different trace output, using NullableType::nullSafeGet() to display information instead of org.hibernate.type tracing. I'm not sure if that is relevant.
Is there something obvious that I am overlooking here? I can't see why I am getting different results when running the same query on the same database.
edit: on OAS I have configured a JDBC ConnectionPool, that uses connection factory class oracle.jdbc.pool.OracleDataSource, and the JDBC data source is a managed data source pointing to that connection pool.
I'm thinking there may be an issue with different Oracle JDBC drivers? The BasicDataSource for the stand-alone app uses the JDBC driver oracle.jdbc.driver.OracleDriver and the dialect org.hibernate.dialect.Oracle10gDialect. I can't see any place in OAS administration that shows the equivalent values.
Please have a look at this article
Looks like, for some reason, OAS returns only 86 bytes of the BLOB value, unless you specify an Lob handler on your configuration.
You can also have more info on this thread of CodeRanch describing the same issue
Hope this helps!

How to configure Dynacache CacheProvider?

I'm trying to implement a Dynacache CacheProvider and having problems. Here is what I've done:
I've got my Dynacache CacheProvider implementation jar under D:\IBM\WebSphere85\AppServer\lib
I have com.ibm.ws.cache.CacheConfig.cacheProviderName configured as JVM custom property with the correct CacheProvider implementation class.
Created cacheinstance.properties located under D:\IBM\WebSphere85\AppServer\properties with the relevant settings including the com.ibm.ws.cache.CacheConfig.cacheProviderName right class name value.
I have the cacheinstance.properties also part of the Dynacache CacheProvider implementation jar.
I have the Object cache Instance configured to have a new dyna cache. This also have the com.ibm.ws.cache.CacheConfig.cacheProviderName as a custom system property.
My application using the following to access the cache:
code:
Properties props = new Properties();
props.put("com.ibm.ws.cache.CacheConfig.cacheProviderName","com.myCacheProvider");
map = (DistributedObjectCache)DistributedObjectCacheFactory.getMap("mycache",props);
I'm getting the following when the application trying to access DynaCache:
[9/18/12 10:10:52:917 EDT] 00000050 ServerCache E DYNA1066E: Unable to initialize the cache provider "com.myCacheProvider". The Dynamic cache will be used to create the cache instance "default" instead of the configured cache provider.
[9/18/12 10:10:52:919 EDT] 00000050 ServerCache E ENGLISH ONLY MESSAGE: cacheProvider is null. Check for the cache provider libraries
[9/18/12 10:10:52:920 EDT] 00000050 ServerCache I DYNA1001I: WebSphere Dynamic Cache instance named default initialized successfully.
I'm using WAS 8.5.
Any ideas what is going on and how to debug this?
Guy,
I would turn on Dyna cache trace to see why this error occurs
Trace String:
com.ibm.ws.cache.=all:com.ibm.ws.drs.=all
This should give us clues on what is happening and depending on what we see from the trace would provide us info on what to do next.
HTH

WebSphere MQ and mmx : Not able to connect with queues

We are using WebSphere MQ and mmx , however we are facing issues while trying to connect with queue:
[2/10/12 13:24:51:861 CST] 00000011 SystemOut O 13:24:51,861 INFO [ListenerThread] - Retry [=1] reconnecting to JMS Queue/Topic
[2/10/12 13:24:51:864 CST] 00000012 SystemOut O 13:24:51,864 INFO [ListenerThread] - Retry [=1] reconnecting to JMS Queue/Topic
[2/10/12 13:24:51:874 CST] 00000012 SystemOut O 13:24:51,874 INFO [JMSListener] - init() failed with JMSException during initializing JMS access: xxsvclnk.queue.ISEEOutboundQueue
[2/10/12 13:24:51:875 CST] 00000011 SystemOut O 13:24:51,875 INFO [JMSListener] - init() failed with JMSException during initializing JMS access: xxreqctr.queue.ISEEInboundQueue
Please let us know possible causes for this issue . we have done all the relevant changes (host name:port ) etc.
Per the update in the comments, the apps don't print linked exceptions and you do not have access to fix them. JMS exceptions are a multi-level data structure. The linked exceptions exist to hold vendor-specific diagnostic codes. If your apps do not print the entire JMS exception then the apps should be reported to the vendor or programmers as containing sev-1 defects. There is no possible justification for a JMS application to not print all levels of a multi-level diagnostic data structure. This post is a brilliant example of exactly why such practices are unjustified. Without a stack trace and linked exceptions, there is no data with which to diagnose production problems. The week (if that) saved on delivery of the code will be paid for many times over with an extended outage.
Your only other option here is to do tracing. Exactly what options are available depends on the versions of WMQ client and WMQ server that are installed. You do have enough access and/or support to find that out, yes?

mqjbnd05 error when deploying app on websphere

I have a fresh install of Wesphere 6.1 Fixpack 23. I have an app deployed that requires an MQSeries JMS Queue. I set up an MQSeries provider-based request and reply queue and an MQSeries provider-based queue connection factory. When the deployed app tries to access the queue, I receive the following error.
Any assistance would be appreciated. Thanks!
[5/28/09 10:33:42:538 EDT] 00000033 ServletWrappe E SRVE0068E: Uncaught exception thrown in one of the service methods of the servlet: espaapp. Exception thrown : org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.UnsatisfiedLinkError: mqjbnd05 (Not found in java.library.path)
Caused by: java.lang.UnsatisfiedLinkError: mqjbnd05 (Not found in java.library.path)
at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:953)
at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:922)
at java.lang.System.loadLibrary(System.java:451)
at com.ibm.mq.MQSESSION.loadLib(MQSESSION.java:1028)
at com.ibm.mq.server.MQSESSION$1.run(MQSESSION.java:246)
at java.security.AccessController.doPrivileged(AccessController.java:192)
at com.ibm.mq.server.MQSESSION.(MQSESSION.java:243)
at java.lang.J9VMInternals.initializeImpl(Native Method)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:177)
at com.ibm.mq.MQSESSIONServer.getMQSESSION(MQSESSIONServer.java:68)
at com.ibm.mq.MQSESSION.getSession(MQSESSION.java:508)
at com.ibm.mq.MQManagedConnectionJ11.(MQManagedConnectionJ11.java:213)
at com.ibm.mq.MQBindingsManagedConnectionFactoryJ11._createManagedConnection(MQBindingsManagedConnectionFactoryJ11.java:186)
at com.ibm.mq.MQBindingsManagedConnectionFactoryJ11.createManagedConnection(MQBindingsManagedConnectionFactoryJ11.java:225)
at com.ibm.mq.StoredManagedConnection.(StoredManagedConnection.java:84)
at com.ibm.mq.MQSimpleConnectionManager.allocateConnection(MQSimpleConnectionManager.java:173)
at com.ibm.mq.MQQueueManagerFactory.obtainBaseMQQueueManager(MQQueueManagerFactory.java:795)
at com.ibm.mq.MQQueueManagerFactory.procure(MQQueueManagerFactory.java:709)
at com.ibm.mq.MQQueueManagerFactory.constructQueueManager(MQQueueManagerFactory.java:664)
at com.ibm.mq.MQQueueManagerFactory.createQueueManager(MQQueueManagerFactory.java:160)
at com.ibm.mq.MQQueueManager.(MQQueueManager.java:550)
at com.ibm.mq.MQSPIQueueManager.(MQSPIQueueManager.java:62)
at com.ibm.mq.jms.MQConnection.createQM(MQConnection.java:2427)
at com.ibm.mq.jms.MQConnection.createQMXA(MQConnection.java:1806)
at com.ibm.mq.jms.MQQueueConnection.(MQQueueConnection.java:105)
at com.ibm.mq.jms.MQQueueConnection.(MQQueueConnection.java:66)
at com.ibm.mq.jms.MQXAQueueConnection.(MQXAQueueConnection.java:59)
at com.ibm.mq.jms.MQXAQueueConnectionFactory.createXAQueueConnection(MQXAQueueConnectionFactory.java:82)
at com.ibm.ejs.jms.JMSManagedQueueConnection.createConnection(JMSManagedQueueConnection.java:123)
at com.ibm.ejs.jms.JMSManagedConnection.(JMSManagedConnection.java:315)
at com.ibm.ejs.jms.JMSManagedQueueConnection.(JMSManagedQueueConnection.java:71)
... More
Does this help?
java.lang.UnsatisfiedLinkError occurs when connecting to a queue manager
Also, within the JMS -> Queue Connection Factories section, select your Queue Connection Factory and check if your "Transport Type" is set to 'BINDINGS' or 'CLIENT'
I swapped mine to CLIENT and that seemed to help a lot.
Such error often happens as a result of passing null to port, host or QManager to connection factory. Try to check all parameters during execution. Normally MQ does not require mqjbnd05 library.
Try to find this file
libmqjbnd05.so
Add that to the LIBPATH for your JVM and try again.
GO to WebSphere Admin console. Environment -> WebSphere variables. Look for MQ_INSTALL_ROOT and modify its value to your MQ installation directory [MQ Home].

Resources