I am using SI configured with a jms:message-driven-channel-gateway . My use case is to receive a message from the queue, save it via JDBC to a database, commit the message from the queue, and then let this message continue to flow through the various channels depending on its type. If the message subsequently errors this is ok as I have the original stored in the database so it can be replayed.
My issue is with trying to commit the transaction from the queue immediately after the database persist. This is effectively mid flow and I have only been able to get the spring transaction management to try and commit right at the end. This is not suitable as if an error is thrown after the database persist it still leaves the message on the JMS queue, as this is where the outer transaction originated.
So, is there an easy way to pull the message from a JMS queue, save to database, then commit it off the queue ?
Thanks!
You need to go asynch after the DB commit - use an ExecutorChannel or a QueueChannel after the DB update; the downstream flow will then run on another thread and the transactions will commit after the handoff.
Related
I am migrating a Spring boot app running on a JEE app server which makes use of JTA to coordinate JMS and JPA transactions:
Exceptions raised while processing a message trigger a JPA and JMS roll-backs (i.e. message goes back to the originating queue)
If all database operations are successful, and, the message is successfully moved to the next queue, JPA and JMS transactions are both committed
The target environment does not support JTA.
I am looking for guidance on how to setup transaction managers so that:
a JPA transaction is started immediately after starting the JMS transaction
a JPA transaction is concluded just before terminating the JMS transaction
a failure of terminating the JPA transaction would fail the JMS transaction
Any documentation or sample code would be awesome.
Many thanks in advance
A possible way forward for this scenario where none of the resources support XA or JTA:
JMS provider
Database driver
... is to use a "bracketing" transaction manager configured to run two transactions. At high level, steps are as specified in Dave Syer's article
Start messaging transaction
Receive message
Start database transaction
Update database
Commit database transaction
Commit messaging transaction
The Spring Data project provides such a transaction manager: see ChainedTransactionManager .
This strategy works well when the error occurs in committing the database transaction, i.e. step 5. For cases where the error happens in the JMS commit, i.e. step 6, one will end up with the message in dead-letter.
Testing this implementation with 100,000 messages showed that about 0.005 % message end up in dead letter. For some, the error occurred when committing the JPA transaction, some for JMS.
For the system to be operable, messages in dead-letter should be retriable regardless of the point of failure. This means that the bracketing transaction manager option is only viable if the app is changed to implement idempotency: keep a trace of the JMS Message IDs already processed. The app has to skip the update part, step 4, for when the JMS Message ID was already processed.
My route looks like below
from("jms:queue:IN_QUEUE) //(A) Transactional Endpoint
.transacted("required") //(B) TX Policy with PROPAGATION_REQUIRED and JPATxManager
.bean("someBean", "readFromDB()") //(C) Read from DB
.bean("someBean", "writeToDB()") //(D) Write to DB
.to("file:/home/src?fileName=demo_${id}.txt")
I know the JMS consumer at (A) will fork out JMS Transaction on each poll and attaches to the
thread.
Also the transacted node in (B) will fork out JPA transaction after an exchange reaches there
and attaches to the thread.
Please find my questions below:
Can two different transactions get attached to a single thread (like the one above) ?
If Yes, which one should get suspended ?
What should be the commit and rollback order of the above mentioned route ?
Note: I didn't find any obvious answer from Camel In Action 2nd Ed book, So please guide me
Good afternoon,
This is a variation on your other question.
The:
from("jms:queue:IN_QUEUE) //(A) Transactional Endpoint
endpoint is transacted, meaning that you have marked the JMS component as transacted and the JMS sessions will be managed by a JmsTransactionManager.
.transacted("required") //(B) TX Policy with PROPAGATION_REQUIRED and
JPATxManager
This should not be a JPA transaction manager, but a JTA transaction manager (like Arjuna). As in your other question, you now have a JMS local transaction for reading you message, and local JPA transacted sessions for your DB access. You want the PlatformTransactionManager (a JTA transaction manager) to synchronize the local transactions for you.
As to your questions:
Can two different transactions get attached to a single thread (like the one above) ?
That really doesn't make any sense.
If Yes, which one should get suspended ?
Nothing will get suspended.
What should be the commit and rollback order of the above mentioned route ?
The DB read is not transactional and does not need to be committed. The file write will actually happen as the JTA transactional context is closed. This leaves the DB write. If that fails, then the DB read does not matter, the message will get put back on the source destination and the file write will not be called.
Enabling the DEBUG logging for the various transaction managers is very helpful.
I could go on about this with this in painful detail. This is more for burki. I think that you will really appreciate this. Very subtle, and it happens a lot.
from("jms:queue:SRC_QUEUE")
.transacted("required")
.to("jms1:queue:DEST_QUEUE")
What happens if the two endpoints are marked as transacted ... but... you do not have the 'transacted' line? Well, a JMS local transaction was started on the message listener. This will be committed as the route ends. There are two independent local JMS transactions. These are not synchronized by a JTA transaction manager.
What actually happens is that the commit for the message 'get' is called. There is no actual commit for the message 'put'. The message 'put' is committed when the JMS session is closed. This is in the JMS spec, that closing the connection inherently commits any transaction. So, because there is no linkage between the two components, the 'get' is committed, and then the 'put' session is closed.
This means that you can lose messages if there is an outage between the commit for the message 'get' and the session close for the message 'put'.
Does that make sense? There is no linkage between the local transactions, so Camel closes them in order, starting with committing the 'get' before calling the 'put'.
JTA transaction synchronization is the key. You still have local transaction resources (not XA), but they can be very well managed in a pretty lightweight JTA transactional context.
from("jms:queue:SRC_QUEUE")
.transacted("required")
.to("DB:transactedwrite")
.to("jms1:queue:DEST_QUEUE")
I couldn't be bothered to look up the correct syntax for a database insert, but you get the idea. In this case, you can get duplicate DB inserts if the JMS 'put' fails. This is not 'all or nothing' XA transactions. The transactions are committed in order. If one in the middle succeeds, then the next transaction fails, well the 'get' will get rolled back and you will get duplicates up to the point of failure.
Sorry, I can't answer your specific questions, but I can give some specific infos about the transactions of your route.
You've got 3 different "systems" with different transaction "scopes"
A JMS broker from which you consume
A database you read and write from and you configured a JPA TxManager for
A file system (no transactions at all) as destination
First of all, if you want to have transaction safety across JMS and the database you have to use XA transactions.
Then, it is unclear if you expect the JMS consumer to be transacted (because of the transacted() in your route) or if you really configured a JMS connection with local JMS transactions. I assume that you really consume transacted.
Let's talk about what you got without line B of your route:
You consume transacted from the broker
Camel processes the message through your route
When any error occurs during route processing, the message is not committed on the broker and therefore redelivered to your route
The transaction that is opened by a transactional consumer is kept open by Camel until the route is successfully processed.
So the only obvious problem would be an error after the database write, that triggers a redelivery and the database write is done once again. Probably the write is not idempotent and therefore must not happen twice.
So to solve these problems, you either have to use XA transactions or you simply use local JMS transactions and implement compensation logic for the "gaps" like the one described above.
The database transaction on the other hand has no benefit unless the read and write operation must be done in a transaction (but I have doubts that this is the case with two individual bean calls and a JMS consumer).
We have what we believe is a fairly common XA use case:
read a message from an in-queue
write some data to a database
write a response message to an out-queue (which is different from the in-queue)
However we also need a way to handle internal errors and poison messages. The control flow we have in mind is something like this:
read the message from the in-queue
write to the database
if there's an exception roll back the database transaction
if there's no exception run commit phase1 on the database
if everything went fine (no rollback and commit phase1 ok) write a success message to the out-queue
if either commit phase1 on the database failed or there was an exception and the database transaction was rolled back write a failure message to the out-queue
commit the in-queue, the out-queue and the database (unless rolled by because of an exception)
Is this a good approach, should we do it differently? How can we do this with EJBs?
We're using EJB 3.1 on JBoss AS 7.2 / EAP 6.1, coding against Narayana directly is an option. The JDBC driver is ojdbc7-12.1.0.1 and the JMS RAR is MQ Series (don't know the version).
What you could do is to use the Java EE event mechanism to get a notification when your transaction fails and create a subsequent output message.
See
(EE7) http://docs.oracle.com/javaee/7/tutorial/doc/cdi-adv005.htm
(EE6) http://docs.oracle.com/javaee/6/tutorial/doc/gkhic.html
You need to use a new transaction to write to the out queue to avoid rolling back the message writing as well.
You will still have the message in the input queue which caused the exception, since the rollback will prevent the successfull consumption. You need to handle that separately, for example by the JMS provider.
I just started using Weblogic JMS. I was able to send messages to the queue and pull them off with a messagebean. Now I want to save the message to a database.
So my question is, how do I tell JMS not to delete the message from the queue until I have successfully written the message to the database?
Thanks
I was able to send messages to the queue and pull them off with a
messagebean.
I suppose you are talking about message-driven bean (MDB)?
So my question is, how do I tell JMS not to delete the message from
the queue until I have successfully written the message to the
database?
MDBs are part of implicit container-managed transaction and the message will not be removed as long as your transaction hasn't commited (that is, as long as your onMessage method hasn't reached its end).
In case of rollback (i.e. you throw an exception or call context.setRollbackOnly() on the MessageDrivenContext), message will be redelivered. You can avoid this behaviour by making transaction bean-managed or using #TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED), but in your situation that should not be the case. Stick with default configuration and everything should work as you wish.
I have a process which involves sending a JMS message.
The process is part of a transaction.
If a later part of the transaction fails, a part that is after a previous part that sent the message, I need to cancel the message.
One thought I had was to somehow set on the message that it is not to be picked up for a certain amount of time, and if I need to rollback, then I could go and cancel the message.
Not knowing messaging, I do not know if the idea is possible.
Or, is there a better idea?
Thanks
You can use JMS and JTA (Java Transaction API) together. When doing that, the sending of a JMS message or the consumption of a received message actually happens atomically as part of the transaction commit.
What does this mean? If the transaction fails or is rolled back, the "sent" message doesn't go out and any "received" messages aren't really consumed. All handled for you by your JMS and JTA provider.
You need to be using a JMS implementation that supports JTA. Sounds like you're already using transactions, so it might be a matter of doing some configuration to make it happen (waving hand vigorously...).
I've had experience using this (BEA WebLogic 7 w/ BEA WebLogic Integration). Worked as advertised -- "the outside world" saw no impact of JMS stuff I tried unless the transaction committed successfully.
Earlier versions of this linked to a Java page describing JMS/JTA integration generally. The page went stale and I don't see an equivalent replacement. This javadoc is for a JMS interface related to this capability.
What you have described is an XA transaction. This allows a transaction to scope across multiple layers i.e. JMS provider, DB or any other EIS. Most containers can be configured to use both non XA and none XA transaction so check your container settings!
For example if you are using JMS with XA transactions the following is possible.
Start Transaction
|
DB Insert
|
Send JMS Msg
|
More DB Inserts
|
Commit Transaction <- Only at this point will the database records be inserted and the JMS message sent.
XA Tranactions are only available in full Java EE containers so XA transactions are not available in Tomcat.
Good luck!
Karl