i'm trying to build a spring boot app that reads messages from kafka and put them into activeMQ
and vice versa (read from activeMQ and write to kafka)
i didn't find any useful tutorial to jumpstart my project
See Spring Integration and the Spring Integration Extension for Apache Kafka.
Use inbound and outbound channel adapters
jms -> kafka
kafka -> jms
Kafka Connect also has some capabilities in this space, but I am not familiar with it.
EDIT
This simple Spring Boot app shows transferring data from Kafka to RabbitMQ and vice versa:
package com.example.demo;
import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.core.QueueBuilder;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.config.TopicBuilder;
import org.springframework.kafka.core.KafkaTemplate;
#SpringBootApplication
public class So61069735Application {
public static void main(String[] args) {
SpringApplication.run(So61069735Application.class, args);
}
#Autowired
private KafkaTemplate<String, String> kafkaTemplate;
#Autowired
private RabbitTemplate rabbitTemplate;
#Bean
public ApplicationRunner toKafka() {
return args -> this.kafkaTemplate.send("so61069735-1", "foo");
}
#KafkaListener(id = "so61069735-1", topics = "so61069735-1")
public void listen1(String in) {
System.out.println("From Kafka: " + in);
this.rabbitTemplate.convertAndSend("so61069735-2", in.toUpperCase());
}
#RabbitListener(queues = "so61069735-2")
public void listen2(String in) {
System.out.println("From Rabbit: " + in);
this.kafkaTemplate.send("so61069735-3", in + in);
}
#KafkaListener(id = "so61069735-3", topics = "so61069735-3")
public void listen(String in) {
System.out.println("Final: " + in);
}
#Bean
public NewTopic topic1() {
return TopicBuilder.name("so61069735-1").partitions(1).replicas(1).build();
}
#Bean
public Queue queue() {
return QueueBuilder.durable("so61069735-2").build();
}
#Bean
public NewTopic topic2() {
return TopicBuilder.name("so61069735-3").partitions(1).replicas(1).build();
}
}
spring.kafka.consumer.auto-offset-reset=earliest
Result
From Kafka: foo
From Rabbit: FOO
Final: FOOFOO
Related
rabbitmq is not creating queue automatically when spring boot publisher send msg ...
i did it this way so it is without manual configuration
and this is my configuration ///////rabbitmq is not creating queue automatically when spring boot publisher send msg ...
i did it this way so it is without manual configuration
and this is my configuration
package com.hariri_stocks.MQ;
import org.springframework.amqp.core.AmqpTemplate;
import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.core.TopicExchange;
import org.springframework.amqp.rabbit.connection.ConnectionFactory;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.amqp.support.converter.Jackson2JsonMessageConverter;
import org.springframework.amqp.support.converter.MessageConverter;
import org.springframework.context.annotation.Bean;
public class msgConfiguration {
public static final String ROUTING_KEY1 = "routingKey1";
public static final String STOCKS_EXCHANGE = "stocks_exchange";
public static final String STOCKS_QUEUE = "stocks_queue";
#Bean
public Queue queue() {
return new Queue(STOCKS_QUEUE , false);
}
#Bean
public TopicExchange exchange() {
return new TopicExchange(STOCKS_EXCHANGE );
}
#Bean
public Binding binding()
{
return BindingBuilder.bind(queue()).to(exchange()).with(ROUTING_KEY1);
}
#Bean
public MessageConverter converter()
{
return new Jackson2JsonMessageConverter();
}
#Bean
public AmqpTemplate template(ConnectionFactory cf) {
final RabbitTemplate rt = new RabbitTemplate(cf);
rt.setMessageConverter(converter());
return rt;
}
}
package com.hariri_stocks.MQ;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
#RestController
public class Givver {
#Autowired
private RabbitTemplate template;
#GetMapping("/msg")
public String send() {
msgStatus m = new msgStatus("ok","damn");
template.convertSendAndReceive(msgConfiguration.STOCKS_EXCHANGE, msgConfiguration.ROUTING_KEY1,m);
return "ok";
}
}
enter image description here
spring.datasource.url=jdbc:mysql://localhost:3306/hariri_stocks
spring.datasource.username=root
spring.datasource.password=
spring.thymeleaf.enabled=true
spring.thymeleaf.check-template-location=true
spring.jpa.hibernate.ddl-auto=update
spring.jpa.generate-ddl=true
spring.jpa.show-sql=true
server.port=9091
spring.application.name=hariri
Class msgConfiguration needs to be annotated with #Configuration for those beans to be added to the application context.
Also you don't need the template bean - Spring Boot will auto wire the converter bean into its auto-configured RabbitTemplate.
I'm having a few issues with Spring Integration and the control bus. I need to turn auto-start off on an InboundChannelAdapter. However when I do this I can't get the ControlBus to start the channel adapter.
I've searched for an answer online, but most of the examples use XML configuration.
Here is the entirety of my code:
package com.example.springintegrationdemo;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.integration.annotation.InboundChannelAdapter;
import org.springframework.integration.annotation.Poller;
import org.springframework.integration.annotation.ServiceActivator;
import org.springframework.integration.channel.DirectChannel;
import org.springframework.integration.config.EnableIntegration;
import org.springframework.integration.config.ExpressionControlBusFactoryBean;
import org.springframework.integration.core.MessageSource;
import org.springframework.integration.file.FileReadingMessageSource;
import org.springframework.messaging.MessageChannel;
import org.springframework.messaging.MessageHandler;
import org.springframework.messaging.support.GenericMessage;
import java.io.File;
#SpringBootApplication
#EnableIntegration
public class SpringIntegrationDemoApplication {
#Bean
public MessageChannel fileChannel() {
return new DirectChannel();
}
#Bean(name = "fileMessageSource")
#InboundChannelAdapter(channel = "fileChannel", poller = #Poller(fixedDelay = "1000"),autoStartup = "false")
public MessageSource<File> fileMessageSource() {
FileReadingMessageSource fileReadingMessageSource = new FileReadingMessageSource();
fileReadingMessageSource.setDirectory(new File("lz"));
return fileReadingMessageSource;
}
#Bean
#ServiceActivator(inputChannel = "fileChannel")
public MessageHandler messageHandler() {
MessageHandler messageHandler = message -> {
File f = (File) message.getPayload();
System.out.println(f.getAbsolutePath());
};
return messageHandler;
}
#Bean
MessageChannel controlChannel() {
return new DirectChannel();
}
#Bean
#ServiceActivator(inputChannel = "controlChannel")
ExpressionControlBusFactoryBean controlBus() {
ExpressionControlBusFactoryBean expressionControlBusFactoryBean = new ExpressionControlBusFactoryBean();
return expressionControlBusFactoryBean;
}
#Bean
CommandLineRunner commandLineRunner(#Qualifier("controlChannel") MessageChannel controlChannel) {
return (String[] args)-> {
System.out.println("Starting incoming file adapter: ");
boolean sent = controlChannel.send(new GenericMessage<>("#fileMessageSource.start()"));
System.out.println("Sent control message successfully? " + sent);
while(System.in.available() == 0) {
Thread.sleep(50);
}
};
}
public static void main(String[] args) {
SpringApplication.run(SpringIntegrationDemoApplication.class, args);
}
}
The message is sent to the control bus component successfully, but the inbound channel adapter never starts.
I would appreciate any help.
Thanks,
Dave
See here: https://docs.spring.io/spring-integration/docs/current/reference/html/configuration.html#annotations_on_beans
The fileMessageSource bean name is exactly for the FileReadingMessageSource. A SourcePollingChannelAdapter created from the InboundChannelAdapter has this bean name: springIntegrationDemoApplication.fileMessageSource.inboundChannelAdapter.
The #EndpointId can help you to simplify it.
In other words: everything is OK with your config, only the problem that you don't use the proper endpoint id to start the SourcePollingChannelAdapter.
I have 2 applications, the first app starts a ActiveMQ broker ( https://spring.io/guides/gs/messaging-jms/ ).
At the second app I want to subcribe a topic from the first app.
How can I do this without starting a ActiveMQ Server?
Possible solution:
Server Application Project
import java.time.LocalDateTime;
import javax.jms.ConnectionFactory;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.activemq.broker.BrokerService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.jms.annotation.EnableJms;
import org.springframework.jms.config.DefaultJmsListenerContainerFactory;
import org.springframework.jms.core.JmsTemplate;
import org.springframework.jms.support.converter.MappingJackson2MessageConverter;
import org.springframework.jms.support.converter.MessageConverter;
import org.springframework.jms.support.converter.MessageType;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.ui.ModelMap;
#SpringBootApplication
#EnableJms
#EnableScheduling
public class JsmServerApplication {
#Autowired
JmsTemplate jmsTemplate;
#Bean
public BrokerService broker() throws Exception {
BrokerService ret = new BrokerService();
ret.addConnector("tcp://0.0.0.0:4444"); // allow remote connections
ret.setBrokerName("primary-broker");
ret.setUseJmx(true);
return ret;
}
#Bean
public ConnectionFactory connectionFactory() {
return new ActiveMQConnectionFactory("tcp://localhost:4444");
}
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory(ConnectionFactory connectionFactory) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setMessageConverter(jacksonJmsMessageConverter());
return factory;
}
#Bean
public MessageConverter jacksonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("_type");
return converter;
}
public static void main(String[] args) {
SpringApplication.run(JsmServerApplication.class, args);
}
#Scheduled(cron = "*/5 * * * * ?")
public void run() {
ModelMap msg = new ModelMap("now", LocalDateTime.now().toString());
System.out.println("Sending: " + msg);
jmsTemplate.convertAndSend("messages", msg);
}
}
Client Application Project
import javax.jms.ConnectionFactory;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.jms.annotation.EnableJms;
import org.springframework.jms.annotation.JmsListener;
import org.springframework.jms.config.DefaultJmsListenerContainerFactory;
import org.springframework.jms.support.converter.MappingJackson2MessageConverter;
import org.springframework.jms.support.converter.MessageConverter;
import org.springframework.jms.support.converter.MessageType;
import org.springframework.ui.ModelMap;
#SpringBootApplication
#EnableJms
public class JsmClientApplication {
#Bean
public ConnectionFactory connectionFactory() {
return new ActiveMQConnectionFactory("tcp://localhost:4444");
}
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory(ConnectionFactory connectionFactory) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setMessageConverter(jacksonJmsMessageConverter());
return factory;
}
#Bean
public MessageConverter jacksonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("_type");
return converter;
}
#JmsListener(destination = "messages", containerFactory = "jmsListenerContainerFactory")
public void msg(ModelMap msg) {
System.out.println(msg);
}
public static void main(String[] args) {
SpringApplication.run(JsmClientApplication.class, args);
}
}
Is it a correct approch?
Solved with this:
http://javasampleapproach.com/java-integration/activemq-work-spring-jms-activemq-topic-publisher-subcribers-pattern-using-springboot
You can use the MessageConsumer to consume the data like the code below
public static void main(String[] args) throws JMSException {
// Getting JMS connection from the server
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory(url);
Connection connection = connectionFactory.createConnection();
connection.start();
Session session = connection.createSession(false,
Session.AUTO_ACKNOWLEDGE);
Topic topic = session.createTopic("topic");
MessageConsumer consumer = session.createConsumer(topic);
MessageListener listner = new MessageListener() {
public void onMessage(Message message) {
try {
//do operations
} catch (JMSException e) {
}
}
};
consumer.setMessageListener(listner);
connection.close();
}
Since you are using the ActiveMQConnectionFactory, you can set the broker as below
BrokerService broker = new BrokerService();
broker.addConnector("tcp://localhost:4444);
broker.setPersistent(false);
ActiveMQConnectionFactory cf = new ActiveMQConnectionFactory("vm://localhost?broker.persistent=false");
If you do not have any restrictions in not using ActiveMQ, You can use Kafka for doing the same. Kafka provides you a highly scalable and distributed Message Bus with simple API.
https://kafka.apache.org/quickstart
I am not sure about the constraints but I just wanted to give you a feel of Kafka. However, the above code should help you in understanding the concept of subscribing and consuming messages from a topic.
See this answer for how to listen on a tcp port instead of the vm:// transport.
i found many code samples to create a JMS queue project but didn't find code sample for activemq( topic) which is called as PUB-SUB(Publisher-Subscriber).
for Spring i found below code but am looking for spring boot complete code .
Topic topic = topicConsumerSession.createTopic("customerTopic");
// Consumer1 subscribes to customerTopic
MessageConsumer consumer1 = topicConsumerSession.createSubscriber(topic);
consumer1.setMessageListener(new ConsumerMessageListener(
"Consumer1"));
// Consumer2 subscribes to customerTopic
MessageConsumer consumer2 = topicConsumerSession.createSubscriber(topic);
consumer2.setMessageListener(new ConsumerMessageListener(
"Consumer2"));
Big Thanks to #Gary Russell first of all.
this is what i have implemented from #Gary Russell suggestions. is there any good practice for separation of concerns and more scalable way.
application.properties
spring.jms.pub-sub-domain=true
#spring.jms.template.default-destination=testTopic
Publisher.java
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.jms.core.JmsTemplate;
import org.springframework.stereotype.Component;
#Component
public class Publisher implements CommandLineRunner{
#Autowired
private JmsTemplate jmsTemplate ;
#Autowired
private Topic topic1;
#Autowired
private Topic topic2;
#Override
public void run(String... arg0) throws Exception {
// TODO Auto-generated method stub
Thread.sleep(5000); // wait for subscriptions, unless they are durable
this.jmsTemplate.convertAndSend(this.topic1,"-----> 1st message from publisher -- topic 1");
Thread.sleep(5000);
this.jmsTemplate.convertAndSend(this.topic1,"-----> 2nd message from publisher -- topic 1");
/**
* for topic2
*/
// TODO Auto-generated method stub
Thread.sleep(5000); // wait for subscriptions, unless they are durable
this.jmsTemplate.convertAndSend(this.topic2,"-----> 1st message from publisher -- topic 2");
Thread.sleep(5000);
this.jmsTemplate.convertAndSend(this.topic2,"-----> 2nd message from publisher -- topic 2");
}
}
Subscriber.java
import org.springframework.jms.annotation.JmsListener;
import org.springframework.stereotype.Component;
#Component
public class Subscriber {
#JmsListener(destination = "Topic1")
public void listener1(String in) {
System.out.println("Listener1: " + in);
}
#JmsListener(destination = "Topic1,Topic2")
public void listener2(String in) {
System.out.println("Listener2: " + in);
}
#JmsListener(destination = "Topic2")
public void listener3(String in) {
System.out.println("Listener3: " + in+"\n listener 3 is just ");
}
}
mainclass : springBootApplication
PubSubJmsBootApplication.java
import javax.jms.Topic;
import org.apache.activemq.command.ActiveMQTopic;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.jms.annotation.EnableJms;
#SpringBootApplication
#EnableJms
public class PubSubJmsBootApplication {
#Bean
public Topic topic1() {
return new ActiveMQTopic("Topic1");
}
#Bean
public Topic topic2() {
return new ActiveMQTopic("Topic2");
}
public static void main(String[] args) {
SpringApplication.run(PubSubJmsBootApplication.class, args);
}
}
you can find spring-boot samples here
In application.properties.
spring.jms.pub-sub-domain=true
spring.jms.template.default-destination=testTopic
Then...
#SpringBootApplication
public class So42173236Application implements CommandLineRunner {
public static void main(String[] args) {
SpringApplication.run(So42173236Application.class, args)
.close();
}
#Autowired
private JmsTemplate jmsTemplate;
#Override
public void run(String... arg0) throws Exception {
Thread.sleep(5000); // wait for subscriptions, unless they are durable
this.jmsTemplate.convertAndSend("foo");
Thread.sleep(5000);
}
#JmsListener(destination = "testTopic")
public void listener1(String in) {
System.out.println("Listener1: " + in);
}
#JmsListener(destination = "testTopic")
public void listener2(String in) {
System.out.println("Listener2: " + in);
}
}
If you build the JmsTemplate or listener container yourself (rather than using Boot's auto configuration), just setPubSubDomain(true).
you can find some easy code in this repo spring-boot-quick
https://github.com/vector4wang/spring-boot-quick/tree/master/quick-activemq
https://github.com/vector4wang/spring-boot-quick/tree/master/quick-activemq2
If you have a lot of news queues and a lot of msg, such as a few million, it is best not to use activemq. I use it in the production environment here, and I am pitted. I need to fill in all kinds of pits. You can use RabbitMQ
In my project created by SpringBoot,
I have added 2 main classes with #SpringBootApplication.
Because if I use STS I can choose one main application when start to debug.
But I found that while SpringDemoApplication is up ,RabbitMQApplication is also running.
Is this specification ? working appropriately?
Here this is sample to reproduce
https://github.com/MariMurotani/SpringDemo/tree/6_rabbitMQ
SpringDemoApplication
package demo;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ApplicationContext;
#SpringBootApplication
public class SpringDemoApplication {
public static void main(String[] args) {
SpringApplication application = new SpringApplication(SpringDemoApplication.class);
ApplicationContext context = application.run(args);
}
}
RabbitMQApplication
package demo;
import java.util.Date;
import org.codehaus.jackson.map.ObjectMapper;
import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.core.TopicExchange;
import org.springframework.amqp.rabbit.connection.ConnectionFactory;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Bean;
import demo.configs.Const;
import demo.dto.Mail;
#SpringBootApplication
public class RabbitMQApplication implements CommandLineRunner {
#Autowired
ApplicationContext context;
#Autowired
RabbitTemplate rabbitTemplate;
#Bean
Queue queue() {
return new Queue(Const.RabbitMQMessageQue, false);
}
#Bean
TopicExchange exchange() {
return new TopicExchange("spring-boot-exchange");
}
#Bean
Binding binding(Queue queue, TopicExchange exchange) {
return BindingBuilder.bind(queue).to(exchange).with(Const.RabbitMQMessageQue);
}
#Bean
SimpleMessageListenerContainer container(ConnectionFactory connectionFactory) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames(Const.RabbitMQMessageQue);
//container.setMessageListener(listenerAdapter);
return container;
}
/*
For asyncronized receiving
#Bean
Receiver receiver() {
return new Receiver();
}
#Bean
MessageListenerAdapter listenerAdapter(Receiver receiver) {
return new MessageListenerAdapter(receiver, "receiveMessage");
}*/
public static void main(String[] args) throws InterruptedException {
SpringApplication.run(RabbitMQApplication.class, args);
}
#Override
public void run(String... args) throws Exception {
System.out.println("Waiting five seconds...");
while(0 < 1){
for(int i = 0 ; i < 5 ; i++){
String object = (String)rabbitTemplate.receiveAndConvert(Const.RabbitMQMessageQue);
if(object != null){
try{
System.out.println(new Date().toGMTString() + ": " + object);
ObjectMapper mapper = new ObjectMapper();
Mail mail = mapper.readValue(object, Mail.class);
System.out.println(mail.getToAddress() + " , " + mail.getStrContent());
}catch(Exception e){
System.out.println(e.getMessage());
}
}
}
Thread.sleep(10000);
}
}
}
The #SpringBootApplication annotation is a shortcut annotation for #Configuration, #EnableAutoConfiguration, and #ComponentScan.
http://docs.spring.io/spring-boot/docs/current/reference/html/using-boot-using-springbootapplication-annotation.html
The default behavior of #ComponentScan is to look for #Configuration and #Component classes within the same package and all sub-packages of the annotated class. Since all your classes are in the same package, when you start any one of them Spring will find the others and treat them like #Configuration classes, and register their beans, etc.
So yes, this is expected behavior given your project setup. Put each #SpringBootApplication class in a separate subpackage if you don't want this to happen for local testing. If this moves beyond a demo at some point you'll probably want to come up with a better setup (subprojects for each #SpringBootApplication perhaps).
I recently faced the same scenario here and I solved with a simple solution.
My projected uses Maven and is configured with sub-modules like this:
my-parent
|__ my-main (depends on my-other module)
|__ my-other
Each module has its own main App class annotated with #SpringBootApplication. The problem is that both classes reside in the same package even though they are in different modules.
Once I start MyMainApp it also starts MyOtherApp. To avoid this I just had to do the following.
In the my-main module I have:
#SpringBootApplication
public class MyMainApp ... { ... }
and in the my-other module I have:
#SpringBootApplication
#ConditionalOnProperty(name = "my.other.active", havingValue = "true", matchIfMissing = false)
public class MyOtherApp ... { ... }
with application.properties with:
my.other.active=true
It works as expected.