I am in the process of developing a stand alone Apache camel application (not running on a J2EE container).
This apps needs to be capable of routing messages from an IBM MQ queue manager to an Oracle database in a distributed transaction.
My google searches pretty much took me to a few places but none of those were able to give me some good clues about how to put everything together.
This link below was the closest to what I need but unfortunately it is not cler enough to put me on the right path.
IBM MQManager as XA Transaction Manager with Spring-jms and Spring-tx
Thank you in advance for your inputs.
You will need to use a JTA TransactionManager, but since not being in a j2ee container i would sugest to use Atomikos.
https://github.com/camelinaction/camelinaction/tree/master/chapter9/xa
My route which is working with IBM MQ -> Oracle Database, in an J2EE yes but still should be working with Atomikos setup. I would say this isn't the proper way to to it, but it's the only way I managed to get it working - working well enough for my use case.
from(inQueue)
.transacted()
.setHeader("storeData", constant(false))
.to("direct:a")
.choice().when(header("storeData").isEqualTo(false)) // if this is true the database calls are 'successful'
.log("Sending message to errorQueue")
.to(errorQueue)
;
StoreDataBean storeDataBean = new StoreDataBean();
from("direct:a")
.onException(Exception.class).handled(false).log("Rollbacking database changes").markRollbackOnlyLast().end()
.transacted("PROPAGATION_REQUIRES_NEW")
.bean(storeDataBean) //storeData sets the header storeData to true if no SQLException or other exceptions are thrown
.end()
;
The commit are handled by the transaction manager, so if I actually get an error with the database commit, it should rollback the message.
Next problem that I have had with rollbacking messages is that I can't set the deadLetterQueue, as you can with ActiveMQ. So it rolls back to the incoming queue. Therefore I actually handle the database transaction as I do, it is in a new transaction, which is rollbacked in case of a normal SQLException when calling the database.
I wished this worked:
from(inQueue)
.onException(Exception.class).to(errorQueue).markRollbackOnly().end()
.bean(storeDataBean)
.end()
I have posted about this in the community forums but no answers at all.
Related
TLDR:
1.Where does atomikos keep its trasnaction records so it can function and how is it best to "secure" them against losing/corrupting them. I use Docker/Docker-Compose and Spring Boot.
2.If the app dies unexpectdly, do all the transactions roll back because atomikos automagically started the appropiate SQL/JMS transactions with the sources?
Details
Hello.
I am starting to use atomikos in Spring Boot to enable XA transactions (in my case simply one datasource and an ActiveMQ/JMS broker.
I have been Googling and can't find a more "in depth" blog or site that explains in a bit more details how Atomikos works under the hood.
I do see the atomikos transaction logs in the form:
{"id":"127.0.1.1.tm159765570032300035","wasCommitted":true,"participants":[{"uri":"127.0.1.1.tm35","state":"COMMITTING","expires":1597655710327,"resourceName":"dataSource"},{"uri":"127.0.1.1.tm36","state":"COMMITTING","expires":1597655710327,"resourceName":"jmsConnectionFactory"}]}
I can only assume these are simply actual "logs" and not like the kafka.db file in say ActiveMQ when using file persistence.
Questions
So does atomikos keep its state in memory?
If say an unexpected shutdown occurred half way through an XA transaction, will all the JTA sources simply rollback?
If the atomikos state is held in a file/database somewhere...are there some best practices when using docker-compose for example?
Example:
A #Component picks out a database entity "REQUIRES_PROCESSING" for queuing on to a JMS queue and then switches the entity to "PROCESSING".
The JMS message is queued but then the App unexpectedly dies before the database entity is set to "PROCESSING".
Will in this case, atomikos has now no state (it died with the app) but it created a SQL and ActiveMQ trasnaction that will simply timeout and everything rolls back?
The application reboots 5 seconds later...what happens here? Is this transaction picked up again somehow (i.e. atomikos has still got state somewhere?).
Will the rebooted app just grab the rolledback entity and simply retry as expected?
Are there any pitfalls here?
Thank you for any claification.
I have recently had to support a colleague in verifying why some system tests are not passing in wildfly, system tests that pass consistently on weblogic and glass fish.
After analysing the log, it became clear the reason is related to a JMS message sent by a backed thread getting committed to a queue too soon, when the expectation was the message would be committed when the entry point Container Managed Transaction of an MDB commits. So the message is going out before the MDB that sends it is done running.
In weblogic, to achieve the expected behaviour, you need to make sure that when you take the connection factory given by the container , which is XA configured, you set the connection.createseesion with
transacted = true and
acknowledgement = session transacted.
In a process similar to the one depicted in this URL
http://www.mastertheboss.com/jboss-server/jboss-jms/sending-jms-messages-over-xa-with-wildfly-jboss-as
Except in the snippet above auto acknowledge is set and the first parameter is set to false.
In wildly when our weblogic and glass fish configuration is used, nothing is committed and the system behaves as if the JMS message sent were to be rolled back.
If configuration as in the example above were to be used, instead what would happen is that the JMS message is immediately and the consumer MDB immediately launches being trigerred before the producer transaction actually ends, causing the system test to fail.
According to the official JMS configuration, by using a connection-pooled factory with the transaction=XA attribute, the container should immediately bind the commit of the transaction to the lifecycle of the parent transaction.
See official documentation bellow in particular in respect to the Java:/JmsXa connection factory.
https://docs.jboss.org/author/display/WFLY10/Messaging+configuration
My colleague was initially using a non pooled connection factory, but the injection info reference has since then been fixed. I have tried all possible combinations of parameters in the shed message, but my outcome is sitll:
Either sent too soon or never sent.
To conclude all the other resources are XA. Namely the oracle db is using the XA driver.
Can anyone confirm if in wildly the send JMS message only when parent transaction commits is working and if so how the session is being configured?
I will check if perhaps my colleague has not made a mistake in terms of the configuration of the connection factory used by the Men's themselves to consume messages out of the queue.but if that one is also XA... Then it is a big problem.
So the issues is fixed.
The commit of the JMS message to the queue at the end of the transaction works perfectly.
The issue was two fold:
(a) First spot of code I was looking at address the issue was not correct. Someone had decided to write his own send telegram to queue API elsewhere, and was not using the central API for sneding telegrams, so any modification I to the injection connection factory was actually not taking effect. The stale connection factories were still being used.
(b) Once the correct API was spotted it was easy to make the mechanism work by using the widlfy XA pooled connection factory mentioned in the post above.
The one thing that was tweaked was the connection.CreationSession api.
The API in JEE 7 has been enlarged and it is now better documented than in jEE 6.
To send a JMS message in a container as part of an XA transaction one should do:
connection.createSession() without any parameters.
This can easily be seen in the connection javadoc:
https://docs.oracle.com/javaee/7/api/javax/jms/Connection.html
QUOTE 1:
This method has been superseded by the method createSession(int
sessionMode) which specifies the same information using a single
argument, and by the method createSession() which is for use in a Java
EE JTA transaction. Applications should consider using those methods
instead of this one.
QUOTE 2:
In a Java EE web or EJB container, when there is an active JTA
transaction in progress:
Both arguments transacted and acknowledgeMode are ignored. The session will participate in the JTA transaction and will be committed
or rolled back when that transaction is committed or rolled back, not
by calling the session's commit or rollback methods. Since both
arguments are ignored, developers are recommended to use
createSession(), which has no arguments, instead of this method.
Which means, the code snippet in:
http://www.mastertheboss.com/jboss-server/jboss-jms/sending-jms-messages-over-xa-with-wildfly-jboss-as
Is not appropriate. What one should be doing is creating the session without any parameter and leting the container handle the rest.
Which it does just fine.
I think I am not getting something right with JMS and JTA. I am running in a Java EE container with all CMTs. Here is what I am doing:
In an SLSB, write something to the database
From the same method of the SLSB, post a message to a JMS queue
An MDB in the same container listens to the JMS queue and picks up the message
The MDB reads the database
The problem is, the MDB does not see the changes made to the database in step 1.
I verified that steps 1 and 2 happen inside a single XA transaction, as expected. My expectation is that a second XA transaction would start at step 3, after the first XA has been committed. But it seems that the MDB receives the message before the XA transaction that posted the message has been committed.
Is my expectation wrong and what I am seeing is normal?
I am running under JBoss 6. The SLSB is local. Both the SLSB and the MDB are in the same application.
I found the problem! My JMS connection factory was not XA aware. I had looked up /XAConnectionFactory for my JMS connection factory. In spite of the name, that's the wrong resource to lookup for a regular app in JBoss. There is a java:/XAConnectionFactory too, which does not work either. The correct resource name is java:/JmsXA. I used it and everything is working just as expected.
Thanks to #strmqm for nudging me to the right direction.
I saw a conceptually similar problem in an app built w/ WebLogic 7. The DB commit from tx1 wasn't complete by the time that tx2 (initiated by a JMS send in tx1) tried to read it.
The trouble there was that our configuration involved a WLS 7 XA emulation layer with a non-XA db connection (to Oracle DB). This risk was part of that XA shortcut. Apparently if we'd gone w/ the true XA all the way to the DB, that hole would have closed. Never ended up testing that.
You say this is JBoss. Any chance that they've got some similar shim that bypasses the XA and gives this same surprising result?
I'm using Oracle 11g for my database and its Oracle Streams AQ feature as JMS implementation.
For all I know, it should be possible to implement a Spring based message-driven POJO (MDP) that uses the same data source for both transactional data access and JMS transactions -- all without XA-Transactions (IIRC, this was marketed as a feature of SpringSource Advanced Pack for Oracle).
Is this possible using Hibernate as well? Ideally, my MDP would start a JMS transaction and read a message from a queue, then re-use the transaction for data access through Hibernate. If anything goes wrong, the JMS and database transaction would both be rolled back, without using 2-phase commit (2PC).
I'm not much of a transaction guru, so before I start digging deeper, can anyone confirm that this is possible and makes sense as well?
Update:
What I want is an implementation of the Shared Transaction Resource pattern. The sample code demonstrates it for ActiveMQ and JDBC, but I need to use Oracle Streams AQ and Hibernate.
Update2:
The SpringSource Advanced Pack for Oracle has been open sourced as part of Spring Data JDBC and it "provides the option of using a single local transaction manager for both
database and message access without resorting to expensive distributed 2-phase commit
transaction management".
2PC shouldn't be necessary, as you say, since the appserver should take care of it. However, you'll pretty much have to use JTA (i.e. JavaEE container) transactions, rather than vanilla DataSource transactions, since JMS only works with JTA.
This isn't a big deal, it's just a bit more fiddly:
Your Spring config should use
<jee:jndi-lookup/> to get a
reference to your container's
DataSource, and you inject that
data source into your spring-managed
hibernate SessionFactory.
You then need to introduce a transaction manager into the context (<tx:jta-transaction-manager/> should work in most app-servers).
In your Spring JMS MessageListenerContainer, plug the above transaction manager reference into it.
Does that all make sense, or should I elaborate? This setup should ensure that the container-managed transactions are held across JMS and Hibernate interactions.
I have a process which involves sending a JMS message.
The process is part of a transaction.
If a later part of the transaction fails, a part that is after a previous part that sent the message, I need to cancel the message.
One thought I had was to somehow set on the message that it is not to be picked up for a certain amount of time, and if I need to rollback, then I could go and cancel the message.
Not knowing messaging, I do not know if the idea is possible.
Or, is there a better idea?
Thanks
You can use JMS and JTA (Java Transaction API) together. When doing that, the sending of a JMS message or the consumption of a received message actually happens atomically as part of the transaction commit.
What does this mean? If the transaction fails or is rolled back, the "sent" message doesn't go out and any "received" messages aren't really consumed. All handled for you by your JMS and JTA provider.
You need to be using a JMS implementation that supports JTA. Sounds like you're already using transactions, so it might be a matter of doing some configuration to make it happen (waving hand vigorously...).
I've had experience using this (BEA WebLogic 7 w/ BEA WebLogic Integration). Worked as advertised -- "the outside world" saw no impact of JMS stuff I tried unless the transaction committed successfully.
Earlier versions of this linked to a Java page describing JMS/JTA integration generally. The page went stale and I don't see an equivalent replacement. This javadoc is for a JMS interface related to this capability.
What you have described is an XA transaction. This allows a transaction to scope across multiple layers i.e. JMS provider, DB or any other EIS. Most containers can be configured to use both non XA and none XA transaction so check your container settings!
For example if you are using JMS with XA transactions the following is possible.
Start Transaction
|
DB Insert
|
Send JMS Msg
|
More DB Inserts
|
Commit Transaction <- Only at this point will the database records be inserted and the JMS message sent.
XA Tranactions are only available in full Java EE containers so XA transactions are not available in Tomcat.
Good luck!
Karl