Is there a mechanism or example implementation of a retry mechanism/solution for a producer using ActiveMQ with JMS (more precisely, with JmsTemplate) and spring framework ?
My use case, which I want to handle is, when the broker is not available, for example, I want to make some number of retries, maximum 6 (if possible with exponential delays between each). So, I need also to track the number of retries for a message between each attempt.
I am aware the the redelivery policy for the consumer, but also I want to implement a reliable producer's client side as well
Thanks,
Simeon
i think that the easiest way is to use what exists for this by using an embedded broker with persistence enabled which must be used by the producer to send the messages to and by creating a Camel route to read from local Queue and forward to the remote one or by using a JmsBridgeConnector or NetworkConnector nut i think the JmsBridgeConnector is easier.
here is an Spring code example :
producer have to use jmsConnectionFactory() to create a ConnectionFactory
package com.example.amq;
import java.io.File;
import javax.jms.ConnectionFactory;
import javax.jms.QueueConnectionFactory;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.activemq.broker.BrokerService;
import org.apache.activemq.network.jms.JmsConnector;
import org.apache.activemq.network.jms.OutboundQueueBridge;
import org.apache.activemq.network.jms.ReconnectionPolicy;
import org.apache.activemq.network.jms.SimpleJmsQueueConnector;
import org.apache.activemq.store.PersistenceAdapter;
import org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
#Configuration
public class ActiveMQConfiguration {
public static final String DESTINATION_NAME = "localQ";
#Bean // (initMethod = "start", destroyMethod = "stop")
public BrokerService broker() throws Exception {
final BrokerService broker = new BrokerService();
broker.addConnector("vm://localhost");
SimpleJmsQueueConnector simpleJmsQueueConnector = new SimpleJmsQueueConnector();
OutboundQueueBridge bridge = new OutboundQueueBridge();
bridge.setLocalQueueName(DESTINATION_NAME);
bridge.setOutboundQueueName("remoteQ");
OutboundQueueBridge[] outboundQueueBridges = new OutboundQueueBridge[] { bridge };
simpleJmsQueueConnector.getReconnectionPolicy().setMaxSendRetries(ReconnectionPolicy.INFINITE);
simpleJmsQueueConnector.setOutboundQueueBridges(outboundQueueBridges);
simpleJmsQueueConnector.setLocalQueueConnectionFactory((QueueConnectionFactory) jmsConnectionFactory());
simpleJmsQueueConnector.setOutboundQueueConnectionFactory(outboundQueueConnectionFactory());
JmsConnector[] jmsConnectors = new JmsConnector[] { simpleJmsQueueConnector };
broker.setJmsBridgeConnectors(jmsConnectors);
PersistenceAdapter persistenceAdapter = new KahaDBPersistenceAdapter();
File dir = new File(System.getProperty("user.home") + File.separator + "kaha");
if (!dir.exists()) {
dir.mkdirs();
}
persistenceAdapter.setDirectory(dir);
broker.setPersistenceAdapter(persistenceAdapter);
broker.setPersistent(true);
broker.setUseShutdownHook(false);
broker.setUseJmx(true);
return broker;
}
#Bean
public QueueConnectionFactory outboundQueueConnectionFactory() {
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(
"auto://localhost:5671");
connectionFactory.setUserName("admin");
connectionFactory.setPassword("admin");
return connectionFactory;
}
#Bean
public ConnectionFactory jmsConnectionFactory() {
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("vm://localhost");
connectionFactory.setObjectMessageSerializationDefered(true);
connectionFactory.setCopyMessageOnSend(false);
return connectionFactory;
}
}
By using Camel :
import org.apache.activemq.camel.component.ActiveMQComponent;
import org.apache.activemq.camel.component.ActiveMQConfiguration;
import org.apache.camel.CamelContext;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;
public class ActiveMQCamelBridge {
public static void main(String args[]) throws Exception {
CamelContext context = new DefaultCamelContext();
context.addComponent("inboundQueue", ActiveMQComponent.activeMQComponent("tcp://localhost:61616"));
ActiveMQComponent answer = ActiveMQComponent.activeMQComponent("tcp://localhost:5671");
if (answer.getConfiguration() instanceof ActiveMQConfiguration) {
((ActiveMQConfiguration) answer.getConfiguration()).setUserName("admin");
((ActiveMQConfiguration) answer.getConfiguration()).setPassword("admin");
}
context.addComponent("outboundQueue", answer);
context.addRoutes(new RouteBuilder() {
public void configure() {
from("inboundQueue:queue:localQ").to("outboundQueue:queue:remoteQ");
}
});
context.start();
Thread.sleep(60 * 5 * 1000);
context.stop();
}
}
Producer does not provide any kind of retry mechanism like consumer. You need to make sure in your code that message sent by producer acknowledge by broker.
Related
What I want to Achieve - Azure Service Bus Message Listener to start / stop receiving messages from queue/topic.
Below is a detailed explanation.
Currently I have integrated Azure Service Bus in my application and we listen message as soon as spring boot application starts. Now I want to modify this logic. By default Azure Service Bus Message Listener will be disable. On ApplicationReadyEvent I want to perform some task and after that again I want to enable Azure Service Bus Message Listener to start listening from topic or queue.
So how can I achieve that ?
application.yml
spring:
cloud:
azure:
servicebus:
namespace: **********
xxx:
azure:
servicebus:
connection: ***********
queue: **********
AzureConfiguration.java
import com.azure.spring.integration.servicebus.inbound.ServiceBusInboundChannelAdapter;
import com.azure.spring.messaging.servicebus.core.ServiceBusProcessorFactory;
import com.azure.spring.messaging.servicebus.core.listener.ServiceBusMessageListenerContainer;
import com.azure.spring.messaging.servicebus.core.properties.ServiceBusContainerProperties;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.integration.channel.DirectChannel;
import org.springframework.integration.dsl.IntegrationFlow;
import org.springframework.integration.dsl.IntegrationFlows;
import org.springframework.messaging.MessageChannel;
#Configuration
public class AzureConfiguration{
#Value("${xxx.azure.servicebus.connection}")
private String serviceBusConnection;
#Value("${xxx.azure.servicebus.queue}")
private String serviceBusQueue;
private static final String SERVICE_BUS_INPUT_CHANNEL = "yyyyy";
private static final String SENSOR_DATA_CHANNEL = "zzzzz";
private static final String SERVICE_BUS_LISTENER_CONTAINER = "aaaaa";
#Bean(name = SERVICE_BUS_LISTENER_CONTAINER)
public ServiceBusMessageListenerContainer serviceBusMessageListenerContainer(ServiceBusProcessorFactory processorFactory) {
ServiceBusContainerProperties containerProperties = new ServiceBusContainerProperties();
containerProperties.setConnectionString(serviceBusConnection);
containerProperties.setEntityName(serviceBusQueue);
containerProperties.setAutoComplete(true);
return new ServiceBusMessageListenerContainer(processorFactory, containerProperties);
}
#Bean
public ServiceBusInboundChannelAdapter serviceBusInboundChannelAdapter(
#Qualifier(SERVICE_BUS_INPUT_CHANNEL) MessageChannel inputChannel,
#Qualifier(SERVICE_BUS_LISTENER_CONTAINER) ServiceBusMessageListenerContainer listenerContainer) {
ServiceBusInboundChannelAdapter adapter = new ServiceBusInboundChannelAdapter(listenerContainer);
adapter.setOutputChannel(inputChannel);
return adapter;
}
#Bean(name = SERVICE_BUS_INPUT_CHANNEL)
public MessageChannel serviceBusInputChannel() {
return new DirectChannel();
}
#Bean(name = SENSOR_DATA_CHANNEL)
public MessageChannel sensorDataChannel() {
return new DirectChannel();
}
#Bean
public IntegrationFlow serviceBusMessageFlow() {
return IntegrationFlows.from(SERVICE_BUS_INPUT_CHANNEL)
.<byte[], String>transform(String::new)
.channel(SENSOR_DATA_CHANNEL)
.get();
}
}
AppEventListenerService.java
import com.azure.spring.integration.servicebus.inbound.ServiceBusInboundChannelAdapter;
import lombok.AllArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Service;
import java.util.List;
#Slf4j
#Service
#AllArgsConstructor
public class AppEventListenerService{
#EventListener(ApplicationReadyEvent.class)
public void OnApplicationStarted() {
log.debug("Enter OnApplicationStarted");
// By Default Azure Service Bus Message Listener will be disable
// do some task
// Enable Azure Bus Message Listener
log.debug("Exit OnApplicationStarted");
}
}
In above code in AppEventListenerService.java ,
// Enable Azure Bus Message Listener - Here I want to start ServiceBusConsumer to receive message from topic/queue.
Literally, if you just want to stop the listener and then start it on ApplicationReadyEvent, then you can autowire the ServiceBusInboundChannelAdapter(or ServiceBusMessageListenerContainer) in your AppEventListenerService.java and then simply call the its stop() and start() API in the AppEventListenerService#OnApplicationStarted method.
However, both the ServiceBusMessageListenerContainer and ServiceBusInboundChannelAdapter implements SmartLifecycle interface and is enabled auto-start-up by default. So if you use the above solution, the listener (as well as adapter) has been triggered to start before ApplicationReadyEvent, which means there will still be a period that the listener is consuming messages.
So I assume you may want to turn off the listener till your own business logic has been done. If so, then currently ServiceBusMessageListenerContainer doesn't provide the function to disable auto-start-up, and we will put your feature request to our backlog.
But you could still use the below workarounds to meet your request.
Workaround-1
You can extend the ServiceBusMessageListenerContainer to override the auto-start-up behavior,
public class CustomServiceBusMessageListenerContainer extends ServiceBusMessageListenerContainer {
private boolean autoStartUp = true;
/**
* Create an instance using the supplied processor factory and container properties.
* #param processorFactory the processor factory.
* #param containerProperties the container properties.
*/
public CustomServiceBusMessageListenerContainer(ServiceBusProcessorFactory processorFactory, ServiceBusContainerProperties containerProperties) {
super(processorFactory, containerProperties);
}
public void setAutoStartUp(boolean autoStartUp) {
this.autoStartUp = autoStartUp;
}
#Override
public final boolean isAutoStartup() {
return this.autoStartUp;
}
}
When declaring the ServiceBusMessageListenerContainer and ServiceBusInboundChannelAdapter bean, disable their auto-start-up function.
#Bean(SERVICE_BUS_LISTENER_CONTAINER)
public ServiceBusMessageListenerContainer messageListenerContainer(ServiceBusProcessorFactory processorFactory) {
ServiceBusContainerProperties containerProperties = new ServiceBusContainerProperties();
containerProperties.setEntityName(QUEUE_NAME);
...
CustomServiceBusMessageListenerContainer listenerContainer = new CustomServiceBusMessageListenerContainer(processorFactory, containerProperties);
listenerContainer.setAutoStartUp(false);
return listenerContainer;
}
#Bean
public ServiceBusInboundChannelAdapter queueMessageChannelAdapter(
#Qualifier(SERVICE_BUS_INPUT_CHANNEL) MessageChannel inputChannel,
#Qualifier(SERVICE_BUS_LISTENER_CONTAINER) ServiceBusMessageListenerContainer listenerContainer) {
ServiceBusInboundChannelAdapter adapter = new ServiceBusInboundChannelAdapter(listenerContainer);
adapter.setOutputChannel(inputChannel);
adapter.setAutoStartup(false);
return adapter;
}
Start the ServiceBusInboundChannelAdapter after your business logic in AppEventListenerService#OnApplicationStarted.
Workaround-2
This might be a bit hack, since we don't expose the api to disable auto-start-up in ServiceBusMessageListenerContainer, but it can be done in ServiceBusInboundChannelAdapter. So you can choose to not declare a bean of ServiceBusMessageListenerContainer but change it as a local variable for the adapter,
#Bean
public ServiceBusInboundChannelAdapter queueMessageChannelAdapter(
#Qualifier(SERVICE_BUS_INPUT_CHANNEL) MessageChannel inputChannel, ServiceBusProcessorFactory processorFactory) {
ServiceBusContainerProperties containerProperties = new ServiceBusContainerProperties();
containerProperties.setEntityName(QUEUE_NAME);
...
ServiceBusMessageListenerContainer listenerContainer = new ServiceBusMessageListenerContainer(processorFactory, containerProperties);
ServiceBusInboundChannelAdapter adapter = new ServiceBusInboundChannelAdapter(listenerContainer);
adapter.setOutputChannel(inputChannel);
adapter.setAutoStartup(false);
return adapter;
}
then start the ServiceBusInboundChannelAdapter after your business logic in AppEventListenerService#OnApplicationStarted.
Here I have a work around where we use the JMS to consume the service bus message
The reason to use JMS is that when we use the #JMSListener we can start stop it.
Now to Implement JMS with ServiceBus refer this MSDOC
Now you have to Autowired this JmsListenerEndpointRegistry object and stop the listener.
#Autowired
JmsListenerEndpointRegistry registry;
To stop the JMS you will have to use the stop function:
registry.stop();
Here I have create two Api which will start/Stop the JMS and receiver of messages:
#Component
#RestController
public class Reciever {
#Autowired
JmsListenerEndpointRegistry registry;
#GetMapping("/stop")
public String readBlobFile ()
{
registry.stop();
return "Stopped" ;
}
#GetMapping("/start")
public String readBlobFile1 ()
{
registry.start();
return "StARTED" ;
}
private static final String QUEUE_NAME = "test";
private final Logger logger = LoggerFactory.getLogger(Reciever.class);
#JmsListener(destination = QUEUE_NAME, containerFactory = "jmsListenerContainerFactory")
public void receiveMessage(String s) {
logger.info("Received message: {}", s);
}
}
Now First I call the /stop Api which will stop the JMS and the message will only start coming once the /startapi is called .
output:
As you're using an integration flow registered as a bean the simplest way to start/stop it is to autowire it as StandardIntegrationFlow and call the corresponding method like so:
#Slf4j
#Service
#DependsOn({"serviceBusMessageFlow"})
#RequiredArgsConstructor
public class AppEventListenerService {
private final StandardIntegrationFlow serviceBusMessageFlow;
#EventListener(ApplicationReadyEvent.class)
public void OnApplicationStarted() {
log.debug("Enter OnApplicationStarted");
// Disable Azure Bus Message Listener
serviceBusMessageFlow.stop();
// do some task
// Enable Azure Bus Message Listener
serviceBusMessageFlow.start();
log.debug("Exit OnApplicationStarted");
}
}
Note the #DependsOn annotation might be needed to force the flow bean to be initialized before the event listener bean.
Also, it should be noted that some messages might happen to go through after the flow is initialized and before the listener is triggered.
I am using RabbitMq(with JMS) with jmsTemplate I am able to Consume Message from RabbitMq Queue But it is taking acknowledgment AUTO.
I have Search API for it but not able to find it out.
How can I set manual acknowledgment.
In Below code when Message is consumed from queue I want to call web service with that message and depends on that response from from I want to delete that message from queue.
I have created one project in which I am using Listener and other project with call to read message from queue
first Project:
package com.es.jms.listener;
import javax.jms.ConnectionFactory;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.TextMessage;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jms.listener.MessageListenerContainer;
import org.springframework.jms.listener.SimpleMessageListenerContainer;
import com.rabbitmq.jms.admin.RMQConnectionFactory;
#Configuration
public class RabbitMqMessageListener {
#Bean
public ConnectionFactory jmsConnectionFactory() {
RMQConnectionFactory connectionFactory = new RMQConnectionFactory();
connectionFactory.setUsername("Username");
connectionFactory.setPassword("Password");
connectionFactory.setVirtualHost("vhostname");
connectionFactory.setHost("hostname");
return connectionFactory;
}
#Bean
public MessageListener msgListener() {
return new MessageListener() {
public void onMessage(Message message) {
System.out.println(message.toString());
if (message instanceof TextMessage) {
try {
String msg = ((TextMessage) message).getText();
System.out.println("Received message: " + msg);
// call web service here and depends on web service
// response
// if 200 then delete msg from queue else keep msg in
// queue
} catch (JMSException ex) {
throw new RuntimeException(ex);
}
}
}
};
}
#Bean
public MessageListenerContainer messageListenerContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(jmsConnectionFactory());
container.setDestinationName("test");
container.setMessageListener(msgListener());
return container;
}
}
2nd Project:
package com.rabbitmq.jms.consumer.controller;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeoutException;
import javax.jms.ConnectionFactory;
import org.json.JSONException;
import org.json.JSONObject;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.jms.JmsException;
import org.springframework.jms.core.JmsTemplate;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.CrossOrigin;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.ResponseBody;
import com.rabbitmq.jms.admin.RMQConnectionFactory;
import redis.clients.jedis.Jedis;
#Controller
public class ReceiverController {
#Autowired
JmsTemplate jmsTemplate;
#Bean
public ConnectionFactory jmsConnectionFactory() {
RMQConnectionFactory connectionFactory = new RMQConnectionFactory();
connectionFactory.setUsername("Username");
connectionFactory.setPassword("Password");
connectionFactory.setVirtualHost("vhostname");
connectionFactory.setHost("hostname");
return connectionFactory;
}
#CrossOrigin
#SuppressWarnings({ "unchecked", "rawtypes" })
#RequestMapping(method = RequestMethod.GET, value = "/getdata")
#ResponseBody
public ResponseEntity<String> fecthDataFromRedis()
throws JSONException, InterruptedException, JmsException, ExecutionException, TimeoutException {
System.out.println("in controller");
jmsTemplate.setReceiveTimeout(500L);
// jmsTemplate.
String message = (String) jmsTemplate.receiveAndConvert("test");
// call web service here and depends on web service
// response
// if 200 then delete msg from queue else keep msg in
// queue
System.out.println(message);
}
return new ResponseEntity(message , HttpStatus.OK);
}
}
How Can I do That?
Thanks In Advance.
You are not using a JmsTemplate, you are using a SimpleMessageListenerContainer to receive the message.
If you were using the template, you would have to use the execute method with a SessionCallback since the acknowledgement must occur within the scope of the session within which the message was received.
However, with the SimpleMessageListenerContainer, you simply set the sessionAcknowledgeMode to Session.CLIENT_ACKNOWLEDGE. See the container javadocs...
/**
* Message listener container that uses the plain JMS client API's
* {#code MessageConsumer.setMessageListener()} method to
* create concurrent MessageConsumers for the specified listeners.
*
* <p>This is the simplest form of a message listener container.
* It creates a fixed number of JMS Sessions to invoke the listener,
* not allowing for dynamic adaptation to runtime demands. Its main
* advantage is its low level of complexity and the minimum requirements
* on the JMS provider: Not even the ServerSessionPool facility is required.
*
* <p>See the {#link AbstractMessageListenerContainer} javadoc for details
* on acknowledge modes and transaction options. Note that this container
* exposes standard JMS behavior for the default "AUTO_ACKNOWLEDGE" mode:
* that is, automatic message acknowledgment after listener execution,
* with no redelivery in case of a user exception thrown but potential
* redelivery in case of the JVM dying during listener execution.
*
* <p>For a different style of MessageListener handling, through looped
* {#code MessageConsumer.receive()} calls that also allow for
* transactional reception of messages (registering them with XA transactions),
* see {#link DefaultMessageListenerContainer}.
...
EDIT
When using the JmsTemplate, you must do your work within the scope of the session - here's how...
First, you have to enable client acknowledge in your template...
this.jmsTemplate.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE);
Then, use the execute method with a SessionCallback ...
Boolean result = this.jmsTemplate.execute(session -> {
MessageConsumer consumer = session.createConsumer(
this.jmsTemplate.getDestinationResolver().resolveDestinationName(session, "bar", false));
String result = null;
try {
Message received = consumer.receive(5000);
if (received != null) {
result = (String) this.jmsTemplate.getMessageConverter().fromMessage(received);
// Do some stuff here.
received.acknowledge();
return true;
}
}
catch (Exception e) {
return false;
}
finally {
consumer.close();
}
}, true);
I am currently setting up a Spring Boot application with Kafka listener.
I am trying to code only the consumer. For producer, I am manually sending message from the Kafka console for now.
I followed the example:
http://www.source4code.info/2016/09/spring-kafka-consumer-producer-example.html
I tried running this as a Spring Boot application but not able to see any messages being received. There are already some messages in my local topic of Kafka.
C:\software\kafka_2.11-0.10.1.0\kafka_2.11-0.10.1.0\kafka_2.11-0.10.1.0\bin\wind
ows>kafka-console-producer.bat --broker-list localhost:9092 --topic test
this is a message
testing again
My Spring Boot application is:
#EnableDiscoveryClient
#SpringBootApplication
public class KafkaApplication {
/**
* Run the application using Spring Boot and an embedded servlet engine.
*
* #param args
* Program arguments - ignored.
*/
public static void main(String[] args) {
// Tell server to look for registration.properties or registration.yml
System.setProperty("spring.config.name", "kafka-server");
SpringApplication.run(KafkaApplication.class, args);
}
}
And Kafka configuration is:
package kafka;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.IntegerDeserializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.EnableKafka;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.config.KafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
import org.springframework.kafka.listener.ConcurrentMessageListenerContainer;
import java.util.HashMap;
import java.util.Map;
#Configuration
#EnableKafka
public class KafkaConsumerConfig {
#Bean
KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory();
factory.setConsumerFactory(consumerFactory());
//factory.setConcurrency(1);
//factory.getContainerProperties().setPollTimeout(3000);
return factory;
}
#Bean
public ConsumerFactory<String, String> consumerFactory() {
return new DefaultKafkaConsumerFactory(consumerConfigs());
}
#Bean
public Map<String, Object> consumerConfigs() {
Map<String, Object> propsMap = new HashMap();
propsMap.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
//propsMap.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
//propsMap.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "100");
//propsMap.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "15000");
propsMap.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class);
propsMap.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
//propsMap.put(ConsumerConfig.GROUP_ID_CONFIG, "group1");
//propsMap.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
return propsMap;
}
#Bean
public Listener listener() {
return new Listener();
}
}
And Kafka listener is:
package kafka;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;
import java.util.concurrent.CountDownLatch;
import java.util.logging.Logger;
public class Listener {
protected Logger logger = Logger.getLogger(Listener.class
.getName());
public CountDownLatch getCountDownLatch1() {
return countDownLatch1;
}
private CountDownLatch countDownLatch1 = new CountDownLatch(1);
#KafkaListener(topics = "test")
public void listen(ConsumerRecord<?, ?> record) {
logger.info("Received message: " + record);
System.out.println("Received message: " + record);
countDownLatch1.countDown();
}
}
I am trying this for the first time. Please let me know if I am doing anything wrong. Any help will be greatly appreciated.
You did not set ConsumerConfig.AUTO_OFFSET_RESET_CONFIG so the default is "latest". Set it to "earliest" so the consumer will receive messages already in the topic.
ConsumerConfig.AUTO_OFFSET_RESET_CONFIG takes effect only if the consumer group does not already have an offset for a topic partition. If you already ran the consumer with the "latest" setting, then running the consumer again with a different setting does not change the offset. The consumer must use a different group so Kafka will assign offsets for that group.
Observed that you dit comment out the consumer group.id property.
//propsMap.put(ConsumerConfig.GROUP_ID_CONFIG, "group1");
Let's see how is quoted in the Kafka official document:
A unique string that identifies the consumer group this consumer belongs to. This property is required if the consumer uses either the group management functionality by using subscribe(topic) or the Kafka-based offset management strategy.
Tried to uncomement that row and the consumer worked.
You will need to annotate your Listener class with either #Service or #Component so that Spring Boot can load the Kafka listener.
package kafka;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;
import java.util.concurrent.CountDownLatch;
import java.util.logging.Logger;
#Component
public class Listener {
protected Logger logger = Logger.getLogger(Listener.class
.getName());
public CountDownLatch getCountDownLatch1() {
return countDownLatch1;
}
private CountDownLatch countDownLatch1 = new CountDownLatch(1);
#KafkaListener(topics = "test")
public void listen(ConsumerRecord<?, ?> record) {
logger.info("Received message: " + record);
System.out.println("Received message: " + record);
countDownLatch1.countDown();
}
}
The above suggestions are good. If you have followed all of them but it did not work, please check if lazy loading is set to false for your application.
The lazy loading is false by default. However if your application had explicit setting like the one below,
spring.main.lazy-initialization=true
Please comment it or make it to false
I am trying to listen to topic using the below snippet. However its listening to queue by default. There is no xml config in this case. I am completely relying on annotations. Moreover I have relied completely on the AutoConfiguration provided by Spring boot. I am not sure how to set the destination type as topic, In JmsListener. Spring JMS gurus please help.
#Component
public class MyTopicListener {
#JmsListener(destination = "${trans.alert.topic}")
public void receiveMessage(TransactionAlert alert) {
logger.info("AlertSubscriberEmail :: Sending Email => <" + alert + ">");
}
}
The answer marked correct is ALMOST correct. It still wont work because:
factory.setPubSubDomain(true)
must come AFTER:
configurer.configure(factory, connectionFactory);
Otherwise the pubSubDomain flag being set to true is lost when configuring the defaults and that factory instance will still work with queues and not topics.
I just took the complete Spring boot example from : https://github.com/spring-guides/gs-messaging-jms/
In this it is created for sending and receipt of messages from a queue. To Change this to a topic , you have to set the Pub-Sub property in the Factory instance.
import org.springframework.beans.factory.support.BeanDefinitionBuilder;
import org.springframework.beans.factory.support.DefaultListableBeanFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.jms.DefaultJmsListenerContainerFactoryConfigurer;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.jms.annotation.EnableJms;
import org.springframework.jms.config.DefaultJmsListenerContainerFactory;
import org.springframework.jms.config.JmsListenerContainerFactory;
import org.springframework.jms.connection.CachingConnectionFactory;
import org.springframework.jms.core.JmsTemplate;
import org.springframework.jms.support.converter.MappingJackson2MessageConverter;
import org.springframework.jms.support.converter.MessageConverter;
import org.springframework.jms.support.converter.MessageType;
import javax.jms.ConnectionFactory;
#SpringBootApplication
#EnableJms
public class JmsSampleApplication {
public void registerBeans(ConfigurableApplicationContext context ){
BeanDefinitionBuilder builder = BeanDefinitionBuilder.rootBeanDefinition(JmsTemplate.class);
CachingConnectionFactory cachingConnectionFactory = new CachingConnectionFactory();
builder.addPropertyValue("connectionFactory", cachingConnectionFactory); // set property value
DefaultListableBeanFactory factory = (DefaultListableBeanFactory) context.getAutowireCapableBeanFactory();
factory.registerBeanDefinition("jmsTemplateName", builder.getBeanDefinition());
}
#Bean
public JmsListenerContainerFactory<?> topicListenerFactory(ConnectionFactory connectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setPubSubDomain(true);
// This provides all boot's default to this factory, including the message converter
configurer.configure(factory, connectionFactory);
// You could still override some of Boot's default if necessary.
return factory;
}
#Bean
public JmsListenerContainerFactory<?> queueListenerFactory(ConnectionFactory connectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
//factory.setPubSubDomain(true);
// This provides all boot's default to this factory, including the message converter
configurer.configure(factory, connectionFactory);
return factory;
}
#Bean // Serialize message content to json using TextMessage
public MessageConverter jacksonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("_type");
return converter;
}
public static void main(String[] args) {
ConfigurableApplicationContext context = SpringApplication.run(JmsSampleApplication.class, args);
JmsTemplate jmsTemplate = context.getBean(JmsTemplate.class);
// Send a message with a POJO - the template reuse the message converter
System.out.println("Sending an email message.");
jmsTemplate.convertAndSend("mailbox.topic", new Email("info#example.com", "Hello"));
jmsTemplate.convertAndSend("mailbox.queue", new Email("info#example.com", "Hello"));
}
}
The listener
package org.springboot.jms;
import org.springframework.jms.annotation.JmsListener;
import org.springframework.stereotype.Component;
/**
* Created by RGOVIND on 10/20/2016.
*/
#Component
public class HelloTopicListener {
#JmsListener(destination = "mailbox.topic", containerFactory = "topicListenerFactory")
public void receiveTopicMessage(Email email) {
System.out.println("Received <" + email + ">");
}
#JmsListener(destination = "mailbox.queue", containerFactory = "queueListenerFactory")
public void receiveQueueMessage(Email email) {
System.out.println("Received <" + email + ">");
}
}
Once this is done , you are all set to subscribe to the topic of choice.
There are multiple approaches to this of course , you can have a map of beans for different jmsTemplates , each of which can be used when you need them based on queue or topic. The template & beans can be instantiated in a method you choose to like discussed in this SO Question. Hope it helps
In Spring Boot's Application.properties, try setting the following property:
spring.jms.pub-sub-domain=true
Then, use this property for the container factory that you are using to listen to the topic.
I am trying to add Spring Integration to a REST MVC Spring app I have been writing. I am using the latest Spring 4.2.x for core, integration and mvc. The idea is to create separate application contexts as on the Dynamic FTP example. The reason why is because I can send emails from 2 separated accounts as well as listen from 2 separated accounts hence having separate application contexts as well as environment variables to aid on bean creation for each context helps a bunch.
I apologize for the newbie questions, but I am having a hard time with the manual as well as trying to figure out how to setup SMTP email configuration class without XML.
I want to have both receive and send integration channels. All email settings will be configured from enviroment variables so I have injected the enviroment: #Autowired Environment env;
I can define:
A MailSender bean
A MailSendingMessageHandler bean
A MessageChannel for the SMTP (outbound)
Now, on XML configurations you have an outbound-channel-adapter where you wire the mail-sender bean as well as the MessageChannel
My goal is to have configurations for:
Send emails.
Listen to IMAP emails and process them.
For sending emails, the idea is to get from a rest endpoint, calling a service and that service is what will put a message to Integration SMTP outbound channel to send an email. Looks like, by using the MailSendingMessageHandler it will get the Integration Message and convert to a Mail Message for the MailSender. I have no idea on how to wire the MailSendingMessageHandler to the outbound channel so that an email can be send. Also I do not know how to, from my #Service class that is called by the rest endpoint how to create the messages and send them through the outbound SMTP channel so emails can be send. On one rest call I send all email recipients I want to reach. Before, each email message body is properly formatted so that I can create each Integration Message (as an email) that will be handled and converted by MailSendingMessageHandler. I have tried to find examples online without success on how to accomplish this.
Any examples you could redirect me? Thanks in advance!
So far I have for the configuration:
import java.util.Properties;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.support.PropertySourcesPlaceholderConfigurer;
import org.springframework.integration.annotation.InboundChannelAdapter;
import org.springframework.integration.config.EnableIntegration;
import org.springframework.integration.annotation.Poller;
import org.springframework.integration.channel.DirectChannel;
import org.springframework.integration.channel.QueueChannel;
import org.springframework.integration.core.MessageSource;
import org.springframework.integration.mail.MailReceiver;
import org.springframework.integration.mail.MailReceivingMessageSource;
import org.springframework.integration.mail.MailSendingMessageHandler;
import org.springframework.mail.MailMessage;
import org.springframework.mail.MailSender;
import org.springframework.mail.javamail.JavaMailSenderImpl;
import org.springframework.messaging.MessageChannel;
import org.springframework.messaging.MessagingException;
import org.springframework.core.env.Environment;
#Configuration
#EnableIntegration
public class IntegrationEmailConfig {
#Autowired
Environment env;
#Bean
public static PropertySourcesPlaceholderConfigurer pspc() {
return new PropertySourcesPlaceholderConfigurer();
}
#Bean
#InboundChannelAdapter(value = "emailInboundChannel", poller = #Poller(fixedDelay = "5000") )
public MailReceivingMessageSource mailMessageSource(MailReceiver imapMailReceiver) {
return new MailReceivingMessageSource(imapMailReceiver);
}
private Properties additionalMailProperties() {
Properties properties = new Properties();
if (env.containsProperty("mail.smtp.auth")) {
properties.setProperty("mail.smtp.auth",env.getProperty("mail.smtp.auth"));
}
if (env.containsProperty("mail.smtp.starttls.enable")) {
properties.setProperty("mail.smtp.starttls.enable",env.getProperty("mail.smtp.starttls.enable"));
}
return properties;
}
#Bean
public MailSender mailSender() throws Exception {
JavaMailSenderImpl mailSender = new JavaMailSenderImpl();
if (env.containsProperty("mail.server.host")) {
mailSender.setHost(env.getProperty("mail.server.host"));
} else {
throw new Exception("Missing mail.server.host property");
}
if (env.containsProperty("mail.server.port")) {
mailSender.setPort(Integer.parseInt(env.getProperty("mail.server.port")));
} else {
throw new Exception("Missing mail.server.port property");
}
if (env.containsProperty("mail.server.username")) {
mailSender.setUsername(env.getProperty("mail.server.username"));
} else {
throw new Exception("Missing mail.server.username property");
}
if (env.containsProperty("mail.server.password")) {
mailSender.setPassword(env.getProperty("mail.server.password"));
} else {
throw new Exception("Missing mail.server.password property");
}
mailSender.setJavaMailProperties(additionalMailProperties());
return mailSender;
}
#Bean
public MailSendingMessageHandler mailSendingMessageHandler() throws Exception {
MailSendingMessageHandler mailSendingMessageHandler = new MailSendingMessageHandler(mailSender());
//mailSendingMessageHandler.setChannelResolver(channelResolver);
return mailSendingMessageHandler;
}
/* #Bean
public DirectChannel outboundMail() {
DirectChannel outboundChannel = new DirectChannel();
return outboundChannel;
}
*/
#Bean
public MessageChannel smtpChannel() {
return new DirectChannel();
}
/* #Bean
#Value("${imap.url}")
public MailReceiver imapMailReceiver(String imapUrl) {
// ImapMailReceiver imapMailReceiver = new ImapMailReceiver(imapUrl);
// imapMailReceiver.setShouldMarkMessagesAsRead(true);
// imapMailReceiver.setShouldDeleteMessages(false);
// // other setters here
// return imapMailReceiver;
MailReceiver receiver = mock(MailReceiver.class);
MailMessage message = mock(Message.class);
when(message.toString()).thenReturn("Message from " + imapUrl);
Message[] messages = new Message[] {message};
try {
when(receiver.receive()).thenReturn(messages);
}
catch (MessagingException e) {
e.printStackTrace();
}
return receiver;
}*/
}
Simply annotate the MailSendingMessageHandler bean with #ServiceActivator, the framework will register a ConsumerEndpointFactoryBean to wrap the handler. See the documentation about "Annotations on #Beans".