I am consuming messages from JMS ActiveMQ using the following code:
<jms:message-driven-channel-adapter
id="helloJMSAdapater" destination="helloJMSQueue" connection-factory="jmsConnectionfactory"
channel="helloChannel" extract-payload="true" />
<integration:channel id="helloChannel" />
My requirement is to consume from here and post it to Kafka outbound adapter. Using the below config:
<int-kafka:outbound-channel-adapter
id="kafkaOutboundChannelAdapter"
kafka-producer-context-ref="kafkaProducerContext"
channel="inputToKafka">
</int-kafka:outbound-channel-adapter>
Here are what i want to achieve:
My queue is a durable topic and dont want to acknowledge the records unless it is successfully published to Kafka. In short, i want to have a transaction behaviour from consuming message from jms to publishing it to Kafka.
I noticed that my messages are immediately dequeued and if processing encounters some exception, i am unable to reprocess it. I dont want that to happen.
Also, when kafka encounters some issue, i want it to be returned to some method so that i can persist the failure message and as said before does not want to acknowledge it.
I am really struggling to get it to work. Can someone please help me out?
You really can have transaction-manager on the <jms:message-driven-channel-adapter> to start TX.
When <int-kafka:outbound-channel-adapter> throws an exception it causes the TX to be ralled back and therefore the message will be requeued.
If you are interested in the persisting errors, there is an error-channel option on the <jms:message-driven-channel-adapter>, but you still have to re-throw exception to let TX to rallback.
To make all that to work you should be sure that there is only single thread from the begging to the end. No <queue> or executor channel in the flow.
Also it isn't clear why do you use so old Apache Kafka still...
Related
I have a service which receives xml messages via an http inbound adapter and then transforms them into text that becomes the content of an email that gets sent out.
I now need to first insert these messages into a JMS queue and send the acknowledgement back as a 200 ok after the message is inserted into the Q and then carry-on with the rest of the processing.
<int-http:inbound-channel-adapter channel="inputChannel"
id="httpInbound"
auto-startup="true"
request-payload-type="java.lang.String"
path="/message"
supported-methods="POST"
error-channel="logger" >
<int-http:request-mapping consumes="application/xml" />
</int-http:inbound-channel-adapter>
<int:chain id="chain" input-channel="inputChannel" >
<int:service-activator ref="mailTransformerBean" method="transform" />
</int:chain>
The service-activator takes care of the processing to convert the xml into an email.
Before that I need to incorporate a JMS Queue into which the received messsages will be inserted and then the acknowledgement is sent back. This is so as to retain the messages and retry in case of a failure of the service.
I would like to set this up as a Transaction with the JMS queue as a endpoint.
How do i approach this?
If you are seeking something like a in-process persistence storage, take a look, please, into the SubscribableJmsChannel :
The channel in the above example will behave much like a normal <channel/> element from the main Spring Integration namespace. It can be referenced by both "input-channel" and "output-channel" attributes of any endpoint. The difference is that this channel is backed by a JMS Queue instance named "exampleQueue".
I have an Apache Camel route configured in Spring which takes a message from a JMS (ActiveMQ) queue, transforms the message and uses the CXF component to send the results to a web service. This all works fine but I always get an exception thrown at the end of the route that the CXF response object isn't synchronizable, referring to the fact that it's trying to convert the exchange/message body back into a JMS message. But why?
Here's my camel context extract:-
<route>
<from uri="jms:queue:transactions" />
<process ref="convertToFormatForCXF" />
<to uri="cxf:bean:myService?defaultOperationName=process" />
<stop />
</route>
and here's a snippet from the logs:-
EndpointMessageListener WARN Execution of JMS message listener failed. Caused by: [org.apache.camel.RuntimeCamelException - java.lang.RuntimeException: net.sophis.soa.dataexchange.LogoutResponse]
org.apache.camel.RuntimeCamelException: java.lang.RuntimeException: net.sophis.soa.dataexchange.LogoutResponse
at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1363)
at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:124)
Even the doesn't prevent the stack trace. Now I've found out that if I add the disableReplyTo=true to the JMS consumer then the stacktrace doesn't show which is excellent.
I suppose I'm just trying to work out what was happening? Was the CXF response object going to be added to the 'transactions' JMS queue? My intention was for the flow to stop once the CXF producer had completed.
Can anyone help my understanding please? The Camel documentation wasn't much help on this.
Was the CXF response object going to be added to the 'transactions' JMS queue?
No. It would have been added to a temporary queue.
From http://camel.apache.org/jms.html#JMS-Request-replyoverJMS
The JmsProducer detects the InOut and provides a JMSReplyTo header with the reply destination to be used. By default Camel uses a temporary queue, but you can use the replyTo option on the endpoint to specify a fixed reply queue (see more below about fixed reply queue).
Camel will automatic setup a consumer which listen on the reply queue, so you should not do anything.
I have the use case that I need to wait 2 hours before consuming messages from an AMQP (we use Rabbit) queue.
EDIT: To clarify my use case... I need each individual message to wait 2 hours before being read. E.g. Message 1 arrives at 10am and Message 2 arrives at 10:15. I need Message 1 to be read at 12p and Message 2 to be read at 12:15p.
We are using Spring Integration 3.x.
The int-amqp:inbound-channel-adapter is message driven and doesn't have a polling option from what I can find.
A couple things I've thought of:
Set auto-startup to false and manually start the inbound channel adapter using a quartz job.
Create my own custom SimpleMessageListenerContainer that is based on polling (not sure how easy this would be)
Configure a "delay queue" in rabbitmq using this method: How to create a delayed queue in RabbitMQ?
EDIT: add 4th option: Use delayer to delay each message for 2 hours: http://docs.spring.io/spring-integration/docs/3.0.2.RELEASE/reference/html/messaging-endpoints-chapter.html#delayer
Any suggestions?
We don't currently have a polling inbound adapter. #1 is easy. For #2, the simplest would be to use a RabbitTemplate and invoke receive() from an inbound-channel-adapter in a POJO.
I would go with #1; you don't need quartz, you can use a simple Spring scheduled task and a control bus to start the adapter.
Another trick is about to use PollableAmqpChannel:
<int-amqp:channel id="myQueueName" message-driven="false"/>
and provide the <poller> for the subscriber to that channel.
There is no reason to send messages to that channel (because you will poll messages from Rabbit Queue) and, right, it looks like anti-pattern, but it is a hook how to avoid any workarounds with direct RabbitTemplate usage via SpEL.
UPDATE
<delayer> can help you, but it depends of your requirements. If you don't want to poll messages from RabbitMQ, you should use the workaround above. But if you just don't want to process message until some time is elapsed, you can just 'delay' it for that time.
Don't forget to add persistent message-store to avoid losing messages during that period and unexpected application failure.
FYI, how I solved the issue. (Used solution #3).
<rabbit:queue name="delayQueue" durable="true">
<rabbit:queue-arguments>
<entry key="x-message-ttl">
<value type="java.lang.Long">7200000</
</entry>
<entry key="x-dead-letter-exchange" value="finalDestinationTopic"/>
<entry key="x-dead-letter-routing-key" value="finalDestinationQueue"/>
</rabbit:queue-arguments>
</rabbit:queue>
<rabbit:topic-exchange name="finalDestinationTopic">
<rabbit:bindings>
<rabbit:binding queue="finalDestinationQueue" pattern="finalDestinationQueue"/>
</rabbit:bindings>
</rabbit:topic-exchange>
My consumer, e.g. service activator that is consuming messages coming from ActiveMQ fromChannel should be restarted when exception occurs or ActiveMQ fails. How to do it for the following spring integration context ?
<!-- RECEIVER. message driven adapter -> jmsInChannel -> activator. -->
<si:channel id="fromChannel"/>
<int-jms:message-driven-channel-adapter id="messageDrivenAdapter"
channel="fromChannel" destination="forward" connection-factory="connectionFactory"
max-concurrent-consumers="2" auto-startup="true" acknowledge="auto" extract-payload="false" />
<si:service-activator id ="activator"
input-channel="fromChannel"
ref="messageService"
method="process"/>
<bean id="messageService" class="com.ucware.ucpo.forward.jms.MessageService"/>
My first idea was to use Retry Advice and add to a service but am not sure if this a right solution for unhandled exceptions. I also would like the receiver to restart if the ActiveMQ server is down.
The listener container within the message-driven-channel-adapter will automatically keep trying to reconnect when it loses connectivity to the broker.
If you set `acknowledge="transacted"' the message will be rolled back on an exception and the broker will resubmit it.
A stateful retry advice would allow you to give up and take some other action after some number of retries (but you can also configure that into ActiveMQ itself where it will send the message to a DLQ after some number of delivery attempts).
Reading your post I instantly thought of this video. Which gives a good insight on how to monitor and control SI application through itself.
Additionally you should have a look at ApplicationEvent documentation of SI.
Glueing that all together you could monitor the JMS message adapter with JMX and stop and restart it through sending an ApplicationEvent on issues. Regarding catching exceptions it depends on what Exceptions you actually want to handle. I'd create an errorChannel that receives exceptions being thrown by components and create a new service that restarts these components after receiving errors.
Following your idea leveraging Spring Retry's capabilites in SI.
I configured spring xml based interceptor, which sends a jms message to activemq queue on each invokation of some transactional method after it is commited. It's happening with the following xml code.
<jms:outbound-channel-adapter channel="filteredStakesChannel" destination="stakesQueue" delivery-persistent="true" explicit-qos-enabled="true" />
But if the activemq server is down i get connection refused exception, which is propagated and i don't want this to happen even if the jms delivery fails. Is this possible?
Should i use some error-channel?
The simplest solution is to make fileredStakesChannel an Executor channel and the send will run on a different thread.
http://static.springsource.org/spring-integration/reference/html/messaging-channels-section.html#executor-channel
http://static.springsource.org/spring-integration/reference/html/messaging-channels-section.html#channel-configuration-executorchannel
Use the <task/> namespace to define an executor to use.