Summary - Message loss during systematic shutdown of application.
I have a application written in spring integration and I am consuming requests from external systems using 'jms:message-driven-channel-adapter'. This is the configuration of the channel adapter -
<jms:message-driven-channel-adapter id="InboundAdapter" destination="InQueue"
connection-factory="connectionFactory"
channel="responseChannel"
error-channel="errorChannel"
acknowledge="transacted"
receive-timeout="50000"
auto-startup="true"/>
my response channel looks like this -
<int:chain id="processorChain" input-channel="responseChannel">
<int:service-activator method="doProcess" ref="inputProcessor" />
<int:router id="inputRouter" method="route">
<int:mapping value="REQUIRED" channel="builderChannel"/>
<int:mapping value="NOT_REQUIRED" channel="TerminateChannel"/>
<bean id="Router" class="xxx.xxx.Router">
</bean>
</int:router>
</int:chain>
Now when i perform 'kill -9 pid', then I see that the message is rolled back to the queue and it is all good. But when I am performing 'kill pid', than the message is lost somewhere. I have enabled teh jms logs and I can see that JMS listener is waiting for the message in transit to complete before closing the JMS consumer but still I don't the message rolling back to my queue. This is the logs snippet that I see in my logs
trace: 15.11.18 15:42:46.619 [Thread-10] DEBUG org.springframework.jms.listener.DefaultMessageListenerContainer - Waiting for shutdown of message listener invokers
trace: 15.11.18 15:42:46.619 [Thread-10] DEBUG org.springframework.jms.listener.DefaultMessageListenerContainer - Still waiting for shutdown of 1 message listener invokers (iteration 0)
After this logs, it is calling the service activators which are defined in the response channel.
Can someone please shed some light on above behavior, I was expecting that when we issue kill command than all messages which are in transit should be rolled back to the queue, instead of this, it is trying to call the activators defined with in channel and after calling last component which is router, it just finishes.
Any help on this topic will be really helpful!!
After some more debugging and banging my head with the system I found the exact scenario where it is happening. Let me put it thru to see if it makes sense. When a systematic shutdown is triggered, JMS is waiting for the messages in transit to finish before stopping the application, till this point everything is good. Currently this chain is being executed -
<int:chain input-channel="inputchain">
<int:transformer id="xxx" method="transform">
<bean class="xxx" />
</int:transformer>
<int:service-activator id="xxx" method="doProcess">
<bean class="xx">
<constructor-arg ref="xxx"/>
</bean>
</int:service-activator>
<int:service-activator id="xxx" ref="rulesProcessor" method="doProcess"/>
<int:service-activator id="xxx" ref="xxx" method="doProcess"/>
<!-- Existing Flow Continues -->
<int:router id="xxxRequiredRouter" method="xxxRequired">
<int:mapping value="Required" channel="firstChannel"/>
<int:mapping value="NotRequired" channel="secondChannel"/>
<bean id="xxxRouter" class="xxx.Router" />
</bean>
</int:router>
</int:chain>
<int:chain input-channel="secondChannel">
some logic
</int:chain>
So the thread finishes this chain and exits gracefully. It doesn't call my next chain 'secondChannel'. My transaction boundary doesn't finish on this chain 'inputchain', it finishes at the end of secondChannel. So I have not comnmitted this transction to database as transaction boundary is at the end of next chain, so this is not available in DB and application is thinking that chain execution is finished so it is complete, so it is not rolled back to teh queue as well. So in teh end I don't have this message in my database and it is not in queue.
So is this the case that only the chain which is being executed when shutdown was triggered will finish and it will not delegate processing to subsequent chains?
A SIGTERM kill will wait for the thread to complete its work.
Eventually the container will interrupt it, but that will only help if it's doing something that's interruptible.
Related
When message Listener in Spring integration is started, it is pulling messages and removing from the source queue, How to stop the delete of messages from source queue?
Make your listener transactional and rollback transaction, when the message from queue will be in your hands:
<int-jms:message-driven-channel-adapter channel="input"
connection-factory="connectionFactory"
transaction-manager="transactionManager"
destination-name="MY-QUEUE"/>
<int:publish-subscribe-channel id="input" />
<int:service-activator input-channel="input" order="1" ref="service" output-channel="output"/>
<int:outbound-channel-adapter channel="input" order="1"
expression="T(org.springframework.transaction.interceptor.TransactionAspectSupport)
.currentTransactionStatus().setRollbackOnly()"/>
But here is need to understand what are you going to do with that remained message in the queue: it becomes available for Listener on the next poll...
I have a production application that seems to be losing messages intermittently. One example that I have identified is the sending system logs 3 messages that are sent milliseconds apart. The receiving system logs that it got message 1 and message 3...but not message 2. The queue's depth is 0. I see no errors in the logs of the application that reads the messages to indicate something bad happened.
I have verified that there are no other "rogue" clients creating a race condition for reading messages.
Here is my configuration; we have a primary and secondary/failover queue...
<bean id="primaryProficiencyInboundQueue" class="com.ibm.mq.jms.MQQueue">
<property name="baseQueueManagerName" value="${lsm.primary.outbound.manager}"/>
<property name="baseQueueName" value="${lsm.proficiency.inbound.queue}"/>
</bean>
<bean id="secondaryProficiencyInboundQueue" class="com.ibm.mq.jms.MQQueue">
<property name="baseQueueManagerName" value="${lsm.secondary.outbound.manager}"/>
<property name="baseQueueName" value="${lsm.proficiency.inbound.queue}"/>
</bean>
<int:channel id="inboundProficiencyChannel">
<int:interceptors>
<int:wire-tap channel="logger" />
</int:interceptors>
</int:channel>
<!-- adapter that connects the inbound proficiency queues to the inboundProficiencyChannel -->
<jms:message-driven-channel-adapter id="primaryProficiencyInboundAdapter"
connection-factory="primaryConnectionFactory"
destination="primaryProficiencyInboundQueue"
channel="inboundProficiencyChannel" />
<jms:message-driven-channel-adapter id="secondaryProficiencyInboundAdapter"
connection-factory="secondaryConnectionFactory"
destination="secondaryProficiencyInboundQueue"
channel="inboundProficiencyChannel" />
<int:service-activator id="proficiencyInboundActivator"
input-channel="inboundProficiencyChannel">
<bean class="com.myactivator.LSMInboundProficiencyActivator" />
</int:service-activator>
Is it possible that something is failing silently?
One other noteworthy thing is that there are two production servers that are both configured as above....meaning that there are two servers that have JMS adapters configured to read the messages. I mentioned above that I see message 1 and message 3 being read. In that case, I see server one process message one and server two process mesage two...so it seems that having two servers configured like above has no negative impact. Any other thoughts on what could be happening or how I can debug the case of the missing/dropped messages?
Version information:
Spring - 3.0.5.RELEASE
Spring Integration - 2.0.3.RELEASE
Nothing springs to mind that would cause lost messages; I don't recall that ever being reported; but 2.0.3 is quite ancient (3rd birthday coming up soon). We don't support 2.0.x any more but you should at least update to 2.0.6 which is the last release of 2.0.x. But preferably move up to the latest 3.0.0 release (or at least 2.2.6).
The issue I have is this:
using service mix and camel routing I am sending a JSON message via activeMQ to consumer.
The issue is that the time that this consumer takes to process this message is X so it is possible the consumer get stopped or crashed during the consuming of the message. In this case the message will be half consumer and will be already deleted from the queue because well it was delivered.
Is it possible to make the queue to not remove messages when they are consumed but instead to wait for some confirmation from the consumer that the processing of this message is done before deleting it?
In a typical importing files from filesystem you will remove the file or rename it to done only at the end ones the file is fully processed and a transaction is fully committed. So how in the ESB world we can say keep the message till I finish and I tell you to remove it.
i am using spring jms:listener-container and jms:listeners for consuming this messages currently.
Your problem is what JMS transactions solves every day.
Some notes from ActiveMQ about it here
You could easily use local transactions in JMS, and setup a listener container like this (note true on sessionTransacted):
<bean id="myListenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="concurrentConsumers" value="1" />
<property name="connectionFactory" ref="jmsConnectionFactory" />
<property name="destination" ref="myQueue" />
<property name="messageListener" ref="myConsumerBean" />
<property name="sessionTransacted" value="true" />
</bean>
Then you have a transacted session. The message will be rolled back to the queue if the message listener fails to consume the message. The transaction will not commit(=message removed from queue) unless the "onMessage" method in the message listener bean returns successfully (no exceptions thrown). I guess this is what you want
I am working on a POC which does the following
Uses a message driven channel adapter to recieve message in a transaction
Calls the Service Activator which uses a handler to insert the message recieved from the adapter to DB and also post message to outbound channel.
Now, if the DB insert of the message fails i want the JMS message returned back to the queue so that it can be re-tried later.
With my below configuration it doesnt seems to work.(i.e. even if there is a failure while inserting into the database the message is removed from the queue.
Any pointers or sample configuration would be helpful.
<integration:channel id="jmsInChannel">
<integration:queue/>
</integration:channel>
<int-jms:message-driven-channel-adapter id="jmsIn"
transaction-manager="transactionManager"
connection-factory="sConnectionFactory"
destination-name="emsQueue"
acknowledge="client" channel="jmsInChannel"
extract-payload="false"/>
<integration:service-activator input-channel="jmsInChannel"
output-channel="fileNamesChannel" ref="handler" method="process" />
<bean id="handler" class="com.irebalpoc.integration.MessageProcessor">
<property name="jobHashTable" ref="jobsMapping" />
</bean>
Set acknowledge="transacted" and, I presume the transactionManager is a JDBC (or JTA) transaction manager.
You also need to remove <queue/> from JmsInChannel so that the database transaction occurs on the same thread.
Spring will synchronize the database transaction with the JMS transaction.
However, read http://www.javaworld.com/javaworld/jw-01-2009/jw-01-spring-transactions.html for the implications.
If you can't make your service idempotent, you may need to look at an XA transaction manager.
we are using Spring Integration 2.1 for persisting messages into database sent by clients.
There is a queue which will be filled by a custom adapter. The configured service activator polls this queue and releases the message to a spring managed #Repository bean. All errors will be captured to an error channel and will be handled by a service. The configuration works so far fine.
My concern is that if the database is not available the service-activators polls all incoming message from the queue and puts them into the error channel. Is there a way to prevent the service-activator to poll the message if the database is obviously not available, for example by sending a test query ?
My configuraton:
<int:channel id="inChannel">
<int:queue />
</int:channel>
<bean id="service" class="some.service.Service" />
<int:service-activator ref="service"
method="write" input-channel="inChannel">
<int:poller fixed-rate="100" task-executor="srvTaskExecutor"
receive-timeout="90" error-channel="errChannel" />
</int:service-activator>
<task:executor id="srvTaskExecutor" pool-size="2-10"
queue-capacity="0" rejection-policy="DISCARD" />
<int:channel id="errChannel" />
<int:service-activator input-channel="errChannel"
ref="errorService" method="write"/>
Regards.
If you give the polling service-activator an "id", you can refer to that instance and call start() or stop() on it based on the DB being available or not. Most likely you'd want to set auto-startup="false" on that service-activator as well.
Additionally, you can even define a "control-bus" element and then send messages like "myActivator.start()" and "myActivator.stop()" to that control bus' input-channel.
Hope that helps,
Mark