spring boot app to integrate kafka with active mq - spring-boot

i'm trying to build a spring boot app that reads messages from kafka and put them into activeMQ
and vice versa (read from activeMQ and write to kafka)
i didn't find any useful tutorial to jumpstart my project

See Spring Integration and the Spring Integration Extension for Apache Kafka.
Use inbound and outbound channel adapters
jms -> kafka
kafka -> jms
Kafka Connect also has some capabilities in this space, but I am not familiar with it.
EDIT
This simple Spring Boot app shows transferring data from Kafka to RabbitMQ and vice versa:
package com.example.demo;
import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.core.QueueBuilder;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.config.TopicBuilder;
import org.springframework.kafka.core.KafkaTemplate;
#SpringBootApplication
public class So61069735Application {
public static void main(String[] args) {
SpringApplication.run(So61069735Application.class, args);
}
#Autowired
private KafkaTemplate<String, String> kafkaTemplate;
#Autowired
private RabbitTemplate rabbitTemplate;
#Bean
public ApplicationRunner toKafka() {
return args -> this.kafkaTemplate.send("so61069735-1", "foo");
}
#KafkaListener(id = "so61069735-1", topics = "so61069735-1")
public void listen1(String in) {
System.out.println("From Kafka: " + in);
this.rabbitTemplate.convertAndSend("so61069735-2", in.toUpperCase());
}
#RabbitListener(queues = "so61069735-2")
public void listen2(String in) {
System.out.println("From Rabbit: " + in);
this.kafkaTemplate.send("so61069735-3", in + in);
}
#KafkaListener(id = "so61069735-3", topics = "so61069735-3")
public void listen(String in) {
System.out.println("Final: " + in);
}
#Bean
public NewTopic topic1() {
return TopicBuilder.name("so61069735-1").partitions(1).replicas(1).build();
}
#Bean
public Queue queue() {
return QueueBuilder.durable("so61069735-2").build();
}
#Bean
public NewTopic topic2() {
return TopicBuilder.name("so61069735-3").partitions(1).replicas(1).build();
}
}
spring.kafka.consumer.auto-offset-reset=earliest
Result
From Kafka: foo
From Rabbit: FOO
Final: FOOFOO

Related

rabbitmq is not creating queue automatically when spring boot publisher send msg

rabbitmq is not creating queue automatically when spring boot publisher send msg ...
i did it this way so it is without manual configuration
and this is my configuration ///////rabbitmq is not creating queue automatically when spring boot publisher send msg ...
i did it this way so it is without manual configuration
and this is my configuration
package com.hariri_stocks.MQ;
import org.springframework.amqp.core.AmqpTemplate;
import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.core.TopicExchange;
import org.springframework.amqp.rabbit.connection.ConnectionFactory;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.amqp.support.converter.Jackson2JsonMessageConverter;
import org.springframework.amqp.support.converter.MessageConverter;
import org.springframework.context.annotation.Bean;
public class msgConfiguration {
public static final String ROUTING_KEY1 = "routingKey1";
public static final String STOCKS_EXCHANGE = "stocks_exchange";
public static final String STOCKS_QUEUE = "stocks_queue";
#Bean
public Queue queue() {
return new Queue(STOCKS_QUEUE , false);
}
#Bean
public TopicExchange exchange() {
return new TopicExchange(STOCKS_EXCHANGE );
}
#Bean
public Binding binding()
{
return BindingBuilder.bind(queue()).to(exchange()).with(ROUTING_KEY1);
}
#Bean
public MessageConverter converter()
{
return new Jackson2JsonMessageConverter();
}
#Bean
public AmqpTemplate template(ConnectionFactory cf) {
final RabbitTemplate rt = new RabbitTemplate(cf);
rt.setMessageConverter(converter());
return rt;
}
}
package com.hariri_stocks.MQ;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
#RestController
public class Givver {
#Autowired
private RabbitTemplate template;
#GetMapping("/msg")
public String send() {
msgStatus m = new msgStatus("ok","damn");
template.convertSendAndReceive(msgConfiguration.STOCKS_EXCHANGE, msgConfiguration.ROUTING_KEY1,m);
return "ok";
}
}
enter image description here
spring.datasource.url=jdbc:mysql://localhost:3306/hariri_stocks
spring.datasource.username=root
spring.datasource.password=
spring.thymeleaf.enabled=true
spring.thymeleaf.check-template-location=true
spring.jpa.hibernate.ddl-auto=update
spring.jpa.generate-ddl=true
spring.jpa.show-sql=true
server.port=9091
spring.application.name=hariri
Class msgConfiguration needs to be annotated with #Configuration for those beans to be added to the application context.
Also you don't need the template bean - Spring Boot will auto wire the converter bean into its auto-configured RabbitTemplate.

Spring Integration how to use Control Bus with JavaConfig, no DSL

I'm having a few issues with Spring Integration and the control bus. I need to turn auto-start off on an InboundChannelAdapter. However when I do this I can't get the ControlBus to start the channel adapter.
I've searched for an answer online, but most of the examples use XML configuration.
Here is the entirety of my code:
package com.example.springintegrationdemo;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.integration.annotation.InboundChannelAdapter;
import org.springframework.integration.annotation.Poller;
import org.springframework.integration.annotation.ServiceActivator;
import org.springframework.integration.channel.DirectChannel;
import org.springframework.integration.config.EnableIntegration;
import org.springframework.integration.config.ExpressionControlBusFactoryBean;
import org.springframework.integration.core.MessageSource;
import org.springframework.integration.file.FileReadingMessageSource;
import org.springframework.messaging.MessageChannel;
import org.springframework.messaging.MessageHandler;
import org.springframework.messaging.support.GenericMessage;
import java.io.File;
#SpringBootApplication
#EnableIntegration
public class SpringIntegrationDemoApplication {
#Bean
public MessageChannel fileChannel() {
return new DirectChannel();
}
#Bean(name = "fileMessageSource")
#InboundChannelAdapter(channel = "fileChannel", poller = #Poller(fixedDelay = "1000"),autoStartup = "false")
public MessageSource<File> fileMessageSource() {
FileReadingMessageSource fileReadingMessageSource = new FileReadingMessageSource();
fileReadingMessageSource.setDirectory(new File("lz"));
return fileReadingMessageSource;
}
#Bean
#ServiceActivator(inputChannel = "fileChannel")
public MessageHandler messageHandler() {
MessageHandler messageHandler = message -> {
File f = (File) message.getPayload();
System.out.println(f.getAbsolutePath());
};
return messageHandler;
}
#Bean
MessageChannel controlChannel() {
return new DirectChannel();
}
#Bean
#ServiceActivator(inputChannel = "controlChannel")
ExpressionControlBusFactoryBean controlBus() {
ExpressionControlBusFactoryBean expressionControlBusFactoryBean = new ExpressionControlBusFactoryBean();
return expressionControlBusFactoryBean;
}
#Bean
CommandLineRunner commandLineRunner(#Qualifier("controlChannel") MessageChannel controlChannel) {
return (String[] args)-> {
System.out.println("Starting incoming file adapter: ");
boolean sent = controlChannel.send(new GenericMessage<>("#fileMessageSource.start()"));
System.out.println("Sent control message successfully? " + sent);
while(System.in.available() == 0) {
Thread.sleep(50);
}
};
}
public static void main(String[] args) {
SpringApplication.run(SpringIntegrationDemoApplication.class, args);
}
}
The message is sent to the control bus component successfully, but the inbound channel adapter never starts.
I would appreciate any help.
Thanks,
Dave
See here: https://docs.spring.io/spring-integration/docs/current/reference/html/configuration.html#annotations_on_beans
The fileMessageSource bean name is exactly for the FileReadingMessageSource. A SourcePollingChannelAdapter created from the InboundChannelAdapter has this bean name: springIntegrationDemoApplication.fileMessageSource.inboundChannelAdapter.
The #EndpointId can help you to simplify it.
In other words: everything is OK with your config, only the problem that you don't use the proper endpoint id to start the SourcePollingChannelAdapter.

How to subcribe to Spring Boot JMS topic from other app

I have 2 applications, the first app starts a ActiveMQ broker ( https://spring.io/guides/gs/messaging-jms/ ).
At the second app I want to subcribe a topic from the first app.
How can I do this without starting a ActiveMQ Server?
Possible solution:
Server Application Project
import java.time.LocalDateTime;
import javax.jms.ConnectionFactory;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.activemq.broker.BrokerService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.jms.annotation.EnableJms;
import org.springframework.jms.config.DefaultJmsListenerContainerFactory;
import org.springframework.jms.core.JmsTemplate;
import org.springframework.jms.support.converter.MappingJackson2MessageConverter;
import org.springframework.jms.support.converter.MessageConverter;
import org.springframework.jms.support.converter.MessageType;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.ui.ModelMap;
#SpringBootApplication
#EnableJms
#EnableScheduling
public class JsmServerApplication {
#Autowired
JmsTemplate jmsTemplate;
#Bean
public BrokerService broker() throws Exception {
BrokerService ret = new BrokerService();
ret.addConnector("tcp://0.0.0.0:4444"); // allow remote connections
ret.setBrokerName("primary-broker");
ret.setUseJmx(true);
return ret;
}
#Bean
public ConnectionFactory connectionFactory() {
return new ActiveMQConnectionFactory("tcp://localhost:4444");
}
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory(ConnectionFactory connectionFactory) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setMessageConverter(jacksonJmsMessageConverter());
return factory;
}
#Bean
public MessageConverter jacksonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("_type");
return converter;
}
public static void main(String[] args) {
SpringApplication.run(JsmServerApplication.class, args);
}
#Scheduled(cron = "*/5 * * * * ?")
public void run() {
ModelMap msg = new ModelMap("now", LocalDateTime.now().toString());
System.out.println("Sending: " + msg);
jmsTemplate.convertAndSend("messages", msg);
}
}
Client Application Project
import javax.jms.ConnectionFactory;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.jms.annotation.EnableJms;
import org.springframework.jms.annotation.JmsListener;
import org.springframework.jms.config.DefaultJmsListenerContainerFactory;
import org.springframework.jms.support.converter.MappingJackson2MessageConverter;
import org.springframework.jms.support.converter.MessageConverter;
import org.springframework.jms.support.converter.MessageType;
import org.springframework.ui.ModelMap;
#SpringBootApplication
#EnableJms
public class JsmClientApplication {
#Bean
public ConnectionFactory connectionFactory() {
return new ActiveMQConnectionFactory("tcp://localhost:4444");
}
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory(ConnectionFactory connectionFactory) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setMessageConverter(jacksonJmsMessageConverter());
return factory;
}
#Bean
public MessageConverter jacksonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("_type");
return converter;
}
#JmsListener(destination = "messages", containerFactory = "jmsListenerContainerFactory")
public void msg(ModelMap msg) {
System.out.println(msg);
}
public static void main(String[] args) {
SpringApplication.run(JsmClientApplication.class, args);
}
}
Is it a correct approch?
Solved with this:
http://javasampleapproach.com/java-integration/activemq-work-spring-jms-activemq-topic-publisher-subcribers-pattern-using-springboot
You can use the MessageConsumer to consume the data like the code below
public static void main(String[] args) throws JMSException {
// Getting JMS connection from the server
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory(url);
Connection connection = connectionFactory.createConnection();
connection.start();
Session session = connection.createSession(false,
Session.AUTO_ACKNOWLEDGE);
Topic topic = session.createTopic("topic");
MessageConsumer consumer = session.createConsumer(topic);
MessageListener listner = new MessageListener() {
public void onMessage(Message message) {
try {
//do operations
} catch (JMSException e) {
}
}
};
consumer.setMessageListener(listner);
connection.close();
}
Since you are using the ActiveMQConnectionFactory, you can set the broker as below
BrokerService broker = new BrokerService();
broker.addConnector("tcp://localhost:4444);
broker.setPersistent(false);
ActiveMQConnectionFactory cf = new ActiveMQConnectionFactory("vm://localhost?broker.persistent=false");
If you do not have any restrictions in not using ActiveMQ, You can use Kafka for doing the same. Kafka provides you a highly scalable and distributed Message Bus with simple API.
https://kafka.apache.org/quickstart
I am not sure about the constraints but I just wanted to give you a feel of Kafka. However, the above code should help you in understanding the concept of subscribing and consuming messages from a topic.
See this answer for how to listen on a tcp port instead of the vm:// transport.

looking for a sample code for springboot, activemq(topic)

i found many code samples to create a JMS queue project but didn't find code sample for activemq( topic) which is called as PUB-SUB(Publisher-Subscriber).
for Spring i found below code but am looking for spring boot complete code .
Topic topic = topicConsumerSession.createTopic("customerTopic");
// Consumer1 subscribes to customerTopic
MessageConsumer consumer1 = topicConsumerSession.createSubscriber(topic);
consumer1.setMessageListener(new ConsumerMessageListener(
"Consumer1"));
// Consumer2 subscribes to customerTopic
MessageConsumer consumer2 = topicConsumerSession.createSubscriber(topic);
consumer2.setMessageListener(new ConsumerMessageListener(
"Consumer2"));
Big Thanks to #Gary Russell first of all.
this is what i have implemented from #Gary Russell suggestions. is there any good practice for separation of concerns and more scalable way.
application.properties
spring.jms.pub-sub-domain=true
#spring.jms.template.default-destination=testTopic
Publisher.java
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.jms.core.JmsTemplate;
import org.springframework.stereotype.Component;
#Component
public class Publisher implements CommandLineRunner{
#Autowired
private JmsTemplate jmsTemplate ;
#Autowired
private Topic topic1;
#Autowired
private Topic topic2;
#Override
public void run(String... arg0) throws Exception {
// TODO Auto-generated method stub
Thread.sleep(5000); // wait for subscriptions, unless they are durable
this.jmsTemplate.convertAndSend(this.topic1,"-----> 1st message from publisher -- topic 1");
Thread.sleep(5000);
this.jmsTemplate.convertAndSend(this.topic1,"-----> 2nd message from publisher -- topic 1");
/**
* for topic2
*/
// TODO Auto-generated method stub
Thread.sleep(5000); // wait for subscriptions, unless they are durable
this.jmsTemplate.convertAndSend(this.topic2,"-----> 1st message from publisher -- topic 2");
Thread.sleep(5000);
this.jmsTemplate.convertAndSend(this.topic2,"-----> 2nd message from publisher -- topic 2");
}
}
Subscriber.java
import org.springframework.jms.annotation.JmsListener;
import org.springframework.stereotype.Component;
#Component
public class Subscriber {
#JmsListener(destination = "Topic1")
public void listener1(String in) {
System.out.println("Listener1: " + in);
}
#JmsListener(destination = "Topic1,Topic2")
public void listener2(String in) {
System.out.println("Listener2: " + in);
}
#JmsListener(destination = "Topic2")
public void listener3(String in) {
System.out.println("Listener3: " + in+"\n listener 3 is just ");
}
}
mainclass : springBootApplication
PubSubJmsBootApplication.java
import javax.jms.Topic;
import org.apache.activemq.command.ActiveMQTopic;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.jms.annotation.EnableJms;
#SpringBootApplication
#EnableJms
public class PubSubJmsBootApplication {
#Bean
public Topic topic1() {
return new ActiveMQTopic("Topic1");
}
#Bean
public Topic topic2() {
return new ActiveMQTopic("Topic2");
}
public static void main(String[] args) {
SpringApplication.run(PubSubJmsBootApplication.class, args);
}
}
you can find spring-boot samples here
In application.properties.
spring.jms.pub-sub-domain=true
spring.jms.template.default-destination=testTopic
Then...
#SpringBootApplication
public class So42173236Application implements CommandLineRunner {
public static void main(String[] args) {
SpringApplication.run(So42173236Application.class, args)
.close();
}
#Autowired
private JmsTemplate jmsTemplate;
#Override
public void run(String... arg0) throws Exception {
Thread.sleep(5000); // wait for subscriptions, unless they are durable
this.jmsTemplate.convertAndSend("foo");
Thread.sleep(5000);
}
#JmsListener(destination = "testTopic")
public void listener1(String in) {
System.out.println("Listener1: " + in);
}
#JmsListener(destination = "testTopic")
public void listener2(String in) {
System.out.println("Listener2: " + in);
}
}
If you build the JmsTemplate or listener container yourself (rather than using Boot's auto configuration), just setPubSubDomain(true).
you can find some easy code in this repo spring-boot-quick
https://github.com/vector4wang/spring-boot-quick/tree/master/quick-activemq
https://github.com/vector4wang/spring-boot-quick/tree/master/quick-activemq2
If you have a lot of news queues and a lot of msg, such as a few million, it is best not to use activemq. I use it in the production environment here, and I am pitted. I need to fill in all kinds of pits. You can use RabbitMQ

Multiple #SpringBootApplication Annotation in a Project

In my project created by SpringBoot,
I have added 2 main classes with #SpringBootApplication.
Because if I use STS I can choose one main application when start to debug.
But I found that while SpringDemoApplication is up ,RabbitMQApplication is also running.
Is this specification ? working appropriately?
Here this is sample to reproduce
https://github.com/MariMurotani/SpringDemo/tree/6_rabbitMQ
SpringDemoApplication
package demo;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ApplicationContext;
#SpringBootApplication
public class SpringDemoApplication {
public static void main(String[] args) {
SpringApplication application = new SpringApplication(SpringDemoApplication.class);
ApplicationContext context = application.run(args);
}
}
RabbitMQApplication
package demo;
import java.util.Date;
import org.codehaus.jackson.map.ObjectMapper;
import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.core.TopicExchange;
import org.springframework.amqp.rabbit.connection.ConnectionFactory;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Bean;
import demo.configs.Const;
import demo.dto.Mail;
#SpringBootApplication
public class RabbitMQApplication implements CommandLineRunner {
#Autowired
ApplicationContext context;
#Autowired
RabbitTemplate rabbitTemplate;
#Bean
Queue queue() {
return new Queue(Const.RabbitMQMessageQue, false);
}
#Bean
TopicExchange exchange() {
return new TopicExchange("spring-boot-exchange");
}
#Bean
Binding binding(Queue queue, TopicExchange exchange) {
return BindingBuilder.bind(queue).to(exchange).with(Const.RabbitMQMessageQue);
}
#Bean
SimpleMessageListenerContainer container(ConnectionFactory connectionFactory) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(Const.RabbitMQMessageQue);
//container.setMessageListener(listenerAdapter);
return container;
}
/*
For asyncronized receiving
#Bean
Receiver receiver() {
return new Receiver();
}
#Bean
MessageListenerAdapter listenerAdapter(Receiver receiver) {
return new MessageListenerAdapter(receiver, "receiveMessage");
}*/
public static void main(String[] args) throws InterruptedException {
SpringApplication.run(RabbitMQApplication.class, args);
}
#Override
public void run(String... args) throws Exception {
System.out.println("Waiting five seconds...");
while(0 < 1){
for(int i = 0 ; i < 5 ; i++){
String object = (String)rabbitTemplate.receiveAndConvert(Const.RabbitMQMessageQue);
if(object != null){
try{
System.out.println(new Date().toGMTString() + ": " + object);
ObjectMapper mapper = new ObjectMapper();
Mail mail = mapper.readValue(object, Mail.class);
System.out.println(mail.getToAddress() + " , " + mail.getStrContent());
}catch(Exception e){
System.out.println(e.getMessage());
}
}
}
Thread.sleep(10000);
}
}
}
The #SpringBootApplication annotation is a shortcut annotation for #Configuration, #EnableAutoConfiguration, and #ComponentScan.
http://docs.spring.io/spring-boot/docs/current/reference/html/using-boot-using-springbootapplication-annotation.html
The default behavior of #ComponentScan is to look for #Configuration and #Component classes within the same package and all sub-packages of the annotated class. Since all your classes are in the same package, when you start any one of them Spring will find the others and treat them like #Configuration classes, and register their beans, etc.
So yes, this is expected behavior given your project setup. Put each #SpringBootApplication class in a separate subpackage if you don't want this to happen for local testing. If this moves beyond a demo at some point you'll probably want to come up with a better setup (subprojects for each #SpringBootApplication perhaps).
I recently faced the same scenario here and I solved with a simple solution.
My projected uses Maven and is configured with sub-modules like this:
my-parent
|__ my-main (depends on my-other module)
|__ my-other
Each module has its own main App class annotated with #SpringBootApplication. The problem is that both classes reside in the same package even though they are in different modules.
Once I start MyMainApp it also starts MyOtherApp. To avoid this I just had to do the following.
In the my-main module I have:
#SpringBootApplication
public class MyMainApp ... { ... }
and in the my-other module I have:
#SpringBootApplication
#ConditionalOnProperty(name = "my.other.active", havingValue = "true", matchIfMissing = false)
public class MyOtherApp ... { ... }
with application.properties with:
my.other.active=true
It works as expected.

Resources