Controlling commit order for a Two-Phase Commit transaction - websphere

Question

We have a java application running on Webshere Application Server 8.5.5. There are two processes running in separate JVMs.
The application is doing some integration using an IBM Webshere Queue Manager and an Oracle 11c database. Both JMS and JDBC resources are configured to use XA transactions.
PROCESS 1:
Read a message from a MESSAGES queue and extract a message key used to identify related messages
Saves message to the database.
Check if the message key does not exist in a KEYS table and if it doesn't it inserts it
ONLY if the insert above happens it puts a message key into a KEYS queue
Commit the transaction
PROCESS 2:
Gets the message key from the KEYS queue
Deletes the message key from the KEYS table
Process all messages associated with received message key from MESSAGES table
What we observed is that in about 5 percent of the cases after getting the message key from the KEYS queue the PROCESS 2 sometimes cannot find the record to delete from the KEYS table just to become available milliseconds later. Trying to delete it multiple times fixed the problem but it became clear that even if PROCESS 1 is XA enabled it commits the JMS resource first and then the JDBC bringing for a very short period of time the data in an inconsistent state.
So here is my question:
Is there a way to force transaction manager to commit JDBC first and then the JMS?
We find this on IBM knowledge center and we try to configure commit-priority like this:
<?xml version="1.0" encoding="UTF-8"?>
<ejb-jar-ext xmlns="http://websphere.ibm.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://websphere.ibm.com/xml/ns/javaee http://websphere.ibm.com/xml/ns/javaee/ibm-ejb-jar-ext_1_0.xsd"
version="1.0">
<session name="RepositoryInstanceFacade">
<resource-ref name="jms/MQConnectionFactory" commit-priority="1" isolation-level="TRANSACTION_READ_COMMITTED"/>
<resource-ref name="jdbc/MessageManagerDB" commit-priority="2" isolation-level="TRANSACTION_READ_COMMITTED"/>
</session>
</ejb-jar-ext>
However this did not solve our problem. I am sure we are not the only ones facing this problem but I just ran out of idea about what to try more. We ended up by putting the message key back in the KEYS queue and that solved the problem but it looks to me like a very unorthodox work around rather than a good solution for a critical 24 x 7 application.
Thank you in advance for your inputs.
UPDATE
I added this resource bindings file ibm-ejb-jar-bnd.xml file in the META-INF folder of my ejb application but still no luck. As part of the deployment I could see IBM manipulated that file and transformed it into an XMI file
<?xml version="1.0" encoding="UTF-8"?>
<ejb-jar-bnd
xmlns="http://websphere.ibm.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://websphere.ibm.com/xml/ns/javaee
http://websphere.ibm.com/xml/ns/javaee/ibm-ejb-jar-bnd_1_0.xsd"
version="1.0">
<session name="RepositoryInstanceFacade">
<resource-ref name="jdbc/MessageManagerDB" binding-name="jdbc/MessageManagerDB"/>
</session>
</ejb-jar-bnd>
UPDATE 2
I ended up with below configuration in two JVMs each JVM running its own application:
application.xml:
<?xml version="1.0" encoding="UTF-8"?>
<application xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/application_6.xsd"
version="6">
...
<resource-ref>
<description>XA Message Manager Data Source</description>
<res-ref-name>java:app/env/jdbc/MessageManagerDB</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>
</application>
ibm-aplication-bnd.xml:
<?xml version="1.0" encoding="UTF-8"?>
<application-bnd
xmlns="http://websphere.ibm.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://websphere.ibm.com/xml/ns/javaee
http://websphere.ibm.com/xml/ns/javaee/ibm-application-bnd_1_0.xsd"
version="1.0">
<resource-ref name="java:app/env/jdbc/MessageManagerDB" binding-name="jdbc/MessageManagerDB"/>
</application-bnd>
ibm-ejb-jar-ext.xml:
<?xml version="1.0" encoding="UTF-8"?>
<ejb-jar-ext xmlns="http://websphere.ibm.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://websphere.ibm.com/xml/ns/javaee http://websphere.ibm.com/xml/ns/javaee/ibm-ejb-jar-ext_1_0.xsd"
version="1.0">
<session name="RepositoryInstanceFacade">
<resource-ref name="java:app/env/jdbc/MessageManagerDB" isolation-level="TRANSACTION_READ_COMMITTED" commit-priority="2"/>
</session>
</ejb-jar-ext>
spring data source bean:
<jee:jndi-lookup id="messageManagerDB" jndi-name="java:app/env/jdbc/MessageManagerDB"/>
And run again my tests. I was very happy during the test run as no warning messages came in the logs. However giving the test was running for a few hours it was hard to manually check all the time. In the end there were still three warning messages showing that JMS message was still read from the queue before the database insert was available. Am I still missing something?
Note that I took out any setting for JMS connection factory as according with the IBM documentation it would stay set to zero so it would have a lower commit priority than JDBC.
UPDATE 3
As advised by the comment and by IBM support I ended up with the following settings below:
ejb-jar.xml:
<session name="RepositoryInstanceFacade">
...
<resource-ref>
<description>XA Message Manager Data Source</description>
<res-ref-name>jdbc/MessageManagerDB</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>
</session>
ibm-ejb-jar-bnd.xml:
<resource-ref name="jdbc/MessageManagerDB" binding-name="jdbc/MessageManagerDB"/>
ibm-ejb-jar-ext.xml:
<session name="RepositoryInstanceFacade">
<resource-ref name="jdbc/MessageManagerDB" isolation-level="TRANSACTION_READ_COMMITTED" commit-priority="2"/>
</session>
spring data source bean:
<jee:jndi-lookup id="messageManagerDB" jndi-name="java:comp/env/jdbc/MessageManagerDB"/>
With these settings in place JMS is still committed first and the JDBC last so it is still not working as expected.

Answer 1

Per the question that I asked in the comments, the resource reference is not actually being used to look up the resource, which is why the commit-priority attribute isn't being honored. In the UPDATE section of the question, you've gotten closer by adding the resource reference binding for the JDBC resource (you'll also need an equivalent for the JMS resource). You will also need to define the resource references in an ejb-jar.xml deployment descriptor (or annotation equivalent if you prefer), and then to look up the resources from your application, you need to use the namespace for the resource reference, which by default is java:comp/env, so you will need to look up the resources as java:comp/env/jms/MQConnectionFactory and java:comp/env/jdbc/MessageManagerDB. If you need visibility outside of the component level, you could would need to define at another scope such as java:module/env or java:app/env which would mean updating the resource reference name to the corresponding value as well.
UPDATE: example of ejb deployment descriptor/bindings/extensions with java:app name
ejb-jar.xml:
<?xml version="1.0" encoding="UTF-8"?>
<ejb-jar xmlns="http://java.sun.com/xml/ns/javaee" version="3.0" metadata-complete="false">
<enterprise-beans>
<session>
<ejb-name>RepositoryInstanceFacade</ejb-name>
<resource-ref>
<description>XA Message Manager Data Source</description>
<res-ref-name>java:app/env/jdbc/MessageManagerDB</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>
</session>
</enterprise-beans>
</ejb-jar>
ibm-ejb-jar-bnd.xml:
<?xml version="1.0" encoding="UTF-8"?>
<ejb-jar-bnd xmlns="http://websphere.ibm.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://websphere.ibm.com/xml/ns/javaee http://websphere.ibm.com/xml/ns/javaee/ibm-ejb-jar-bnd_1_0.xsd"
version="1.0">
<session name="RepositoryInstanceFacade">
<resource-ref name="java:app/env/jdbc/MessageManagerDB" binding-name="jdbc/MessageManagerDB"/>
</session>
</ejb-jar-bnd>
ibm-ejb-jar-ext.xml:
<?xml version="1.0" encoding="UTF-8"?>
<ejb-jar-ext xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://websphere.ibm.com/xml/ns/javaee"
xsi:schemaLocation="http://websphere.ibm.com/xml/ns/javaee http://websphere.ibm.com/xml/ns/javaee/ibm-ejb-jar-ext_1_0.xsd"
version="1.0">
<session name="RepositoryInstanceFacade">
<resource-ref name="java:app/env/jdbc/MessageManagerDB" isolation-level="TRANSACTION_READ_COMMITTED" commit-priority="2"/>
</session>
</ejb-jar-ext>
Application code to look up the data source and verify that the isolation level from the extensions is being used:
DataSource ds = InitialContext.doLookup("java:app/env/jdbc/MessageManagerDB");
Connection con = ds.getConnection();
try {
int iso = con.getTransactionIsolation();
System.out.println("Isolation level is " + iso);
} finally {
con.close();
}
If the above code outputs 2 (the value of java.sql.Connection.TRANSACTION_READ_COMMITTED), rather than the default isolation level of 4, this demonstrates that the bindings/extensions are being applied.
If after this point, you still don't seeing the commit-priority being honored, then it's a transaction management bug and I recommend reporting it to IBM support.