I am new to using spring integration kafka. I was reading the docs, and got confused between Inbound channel adapter and message driven channel adapters. Can some one explain the difference between them . Also does spring integration for kafka has some error handling mechanism like the one present in spring integration for amqp/rmq( using dlq).
PS : We are trying to move away from rabbit mq to kafka as source of messages in ithe spring integration framework
thanks,
The message-driven adapter is similar to the rabbitmq inbound adapter in that messages are "pushed" into your integration flow whenever records are available in the topic.
The inbound channel adapter uses the "pull" model, where you poll for messages on a schedule. There is an equivalent for RabbitMQ (since 5.0.1) but it's not currently configurable with XML.
some error handling mechanism
The SIK components are subclasses of the same classes as all other SI components, so the same facilities are generally available, yes.
Related
I am currently trying to write an adapter which will consume messages from ActiveMQ and publish it to Kafka.
I am thinking of using spring integration to integrate these two messaging systems.
My problem is that my application will not maintain registry of the Models using which many applications will publish the records to activeMQ. I want to receive these javax jms message and want to perform some transformation like adding jmscorrelationId into kafka message.
ALso, another requirement is to send acknowledgement to active mq only when kafka send/publish is successfull.
Can ack be send back to activemq using spring integration?
Will spring integration be a good option?
Kindly note my tech architect is not in favor of using Camel/Mule. Also, he does not want to use Kafka Connect as i was planning to use Kafka connect source.
Please suggest.
The Spring Integration Kafka extension project has a sync mode for publishing, which will block the thread until Kafka confirms delivery (or throw an exception on a failure).
The JMS inbound gateway can be used to return a reply to a JMS queue.
You can add transformers (or whatever) in the flow to modify the message.
Can someone explain different ways of configuring message listener.
I know two ways:
Spring Jms Listener
EJB MDB way.
Are there any other ways (should be applicable to both IBM MQ and Active MQ)?
For the first question, your proposed ways are good ones with Camel JMS.
For the second question take a look at Java JMS mix messaging implementations
if you want to use the same client without changing anything you have to use AMQP ptotocol wich is designed for this.
here is 2 examples :
ActiveMQ AMQP with JMS transformer leveraging spring Integration
Unable to access ActiveMQ using JMS based code and amqp 1.0
Can I send a message to a message broker using SimpMessagingTemplate#convertAndSendToUser or SimpMessagingTemplate#convertAndSend methods without settings up a websocket message broker using #EnableWebSocketMessageBroker?
What I'm trying to do is utilise one websocket server to provide messaging for two application server instances(One spring 4 and one Spring 3). I created a one web server with Spring 4, Spring boot plus websocket message broker enabled.
Now I want two application servers to push messages to rabbitmq so it will broadcast them to clients subscribed to it.
First issue I faced is if there is no websockt message broker configuration available, SimpMessagingTemplate will not get autowired to application context. I couldn't get it injected without creating a websocket message board either.
Please help me to find out whether this is possible.
BTW I have a previous question unanswered related to this.
Well, After reading lots of documentation I found the answer myself. The key thing is this architecture is following.
In this architecture spring act as a gateway for communication between the message broker and client. Spring doesn't do anything(Other than when it necessary) but forward the request to the message broker(STOMP messages). The configuration kept on Spring defines couple of important things. One is the exchange and other were routing keys. Spring configuration gives us an abstract layer so we subscribe and push messages to message broker without a fuss.
SimpMessagingTemplate is the abstract layer which we use to communicate with message broker. Spring creates the bean using the given details. Well I couldn't create a instance of SimpMessagingTemplate manually. I have to update Spring 3 application to Spring 4 in order to use websockets.
Since Spring and message broker is decoupled, clustering the application instance doesn't make any effect on message broker. Spring will communicate to message broker only when it need to subscribe to a channel or when it need to publish a message to a channel. So if there is two instances subscribing to same channel it would be two queues binding the one exchange using same routing key. Messages published into a channel will be available to all subscribers(queues) because they all use same routing key. Refer to rabbitmq stop plugin documentation for more elaborative description.
I am using Spring, Spring-Websocket, STOMP for my application, and RabbitMQ as broker. I need to log all messages going through RabbitMQ to Postgresql tables.
I know that I can write #MessageMapping in Spring and log there, but my problem is that some clients talk to RabbitMQ directly through MQTT protocol, and Spring does not support it yet (https://jira.spring.io/browse/SPR-12581). Moreover browser clients talk through Spring to RabbitMQ using STOMP protocol.
RabbitMQ allows to track all messages using Firehose tracer. How to properly listen to amq.rabbitmq.trace topic from Spring? Or do I need to write separate Java app as consumer?
The Spring AMQP is for you!
You bind some custom queue to to that amq.rabbitmq.trace with appropriate pattern (e.g. publish.#) and configure SimpleMessageListenerContainer to receive messages from that queue.
It can be done even with pretty simple config: #EnableRabbit and #RabbitListener on some POJO method. Anyway the Binding #Bean must be there to attache your queue to that exchange.
Application Data Flow:
JSon Messages--> Active MQ --> Spring XD-- Business Login(Transform JSon to Java Object)--> Save Data to Target DB--> DB.
Question:
Sprin-Xd is running in cluster mode, configured with Radis.
Spring XD picks up the message from the Active message queue(AMQ). So message is no longer in AMQ. Now while one of the containers where this message is being processed with some business logic suddenly goes down. In this scenarios-
Will Spring-XD framework automatically re-process that particular message ? what's mechanism behind that?
Thanks,
Abhi
Not with a Redis transport; Redis has no infrastructure to support such a requirement ("transactional" reads). You would need to use a rabbit or kafka transport.
EDIT:
See Application Configuration (scroll down to RabbitMQ) and Message Bus Configuration.
Specifically, the default ackMode is AUTO which means messages are acknowledged on success.