How can one ensure in Spring Cloud Stream Rabbit guaranteed delivery. My code below : -
class Source {
MessageChannel output;
Repository repo;
#Transactional
void publisher(Command command){
repo.save(command);
output.send(MessageBuilder
.withPayload(new Event()).build());
}
}
class Sink {
#StreamListener(Event.class)
void eventListener(Event event){
// method body
}
}
Any help is appreciated.
You can use the Rabbit consumer property spring.cloud.stream.rabbit.bindings.<channelName>.consumer.acknowledgeMode on how do you want to acknowledge. The acknowledgeMode comes from Spring AMQP and you can refer more documentation on this here
Related
i am writing a microservices application based on kafka request/reply semantic, so i got configured ReplyingKafkaTemplate as message producer and #KafkaListener with #SendTo annotations method as the service request listener. I need to create a class with method marked with #KafkaListeners and #SendTo annotations dynamically based on topic name. It should be something like this:
#Component
class KafkaReceiver {
// LISTENER FOR OTHER MICROSERVICE REQUESTS
#KafkaListener("#{topicProvider.getTopic()})
#SendTo
public Response listen(Request request) {
... some logic
return response;
}
}
#Component
class KafkaRecevierFactory {
public KafkaReceiver createListener(String topic) {
...
return kafkaReceiver;
}
//OR SOMETHING LIKE THIS:
public void runListenerContainer(String topic, ReqeustProcessor processor) {
Container container = ContainerFactory.create(topic)
container.setListener( request -> {
Response resp = processor.process(request);
return resp;
});
container.start();
}
}
Is there anyway i can do this?
I am using Spring Boot's #KafkaListener to monitor some server's heartbeat messages as:
#KafkaListener(topics = "heartbeat-topic", groupId = "monitor")
public void listenToHeartbeatMsg(String message) {}
Issue is when the listener subscriber application started, even though the server is down, subscriber application will be receiving those previous heartbeat messages.
How can I fix this issue and listen only for real-time heartbeat messages?
Implement ConsumerSeekAware and, in onPartitionsAssigned call seekToBeginning on the callback.
See https://docs.spring.io/spring-kafka/docs/current/reference/html/#seek
public class MyListener implements ConsumerSeekAware {
...
#Override
public void onPartitionsAssigned(Map<TopicPartition, Long> assignments, ConsumerSeekCallback callback) {
callback.seekToEnd(assignments.keySet());
}
}
I'm very new with Kafka.
Using spring-boot kafka, I developed a publisher and a consumer using one Message object and manual ack. My code uses spring annotation. That's works perfectly.
Now, when I connect to production brokers, this one not send one Message but a list of message.
My listener method has the following signature:
#KafkaListener (topics="MessagesTopic", containerFactory="messageContainerfactory")
public void listen(#Payload Message message, Acknowledgment ack)
so I can acknowledge each Message. Good.
But now it's seems I must replace it with
#KafkaListener (topics="MessagesTopic", containerFactory="messageContainerfactory")
public void listen(#Payload List<Message> messages, Acknowledgment ack)
Even following the documentation it seems that I should use
#KafkaListener (topics="MessagesTopic", containerFactory="messageContainerfactory")
public void listen(#Payload List<Message> messages, Acknowledgment ack, Consumer<?,?> consumer)
Should I set batchmode to true ?
Now the question is : how can I acknowledge each message when this one has been completely handled?
Many many thanks for your help
Something like this one can help you either if you do want to manually commit offset.
If you do not want it then switch setAckMode to other value.
Here's this thing done the spring-way.
CoreAutoConfiguration class:
#Configuration
#Import({KafkaAutoConfiguration.class})
public class CoreAutoConfiguration {
#Bean("batchKafkaListenerContainerFactory")
public ConcurrentKafkaListenerContainerFactory<?, ?> batchKafkaListenerContainerFactory(ConcurrentKafkaListenerContainerFactoryConfigurer configurer, ConsumerFactory<Object, Object> kafkaConsumerFactory) {
ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
configurer.configure(factory, kafkaConsumerFactory);
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
factory.setBatchListener(true);
return factory;
}
}
Then there goes your Config class:
#Configuration
#Import({
CoreAutoConfiguration.class,
KafkaAutoConfiguration.class,
})
#EnableKafka
#EnableRetry
public class Config {
}
Finally the consumer:
#KafkaListener(
topics = "MessagesTopic",
containerFactory = "batchKafkaListenerContainerFactory"
)
public void dataReceived(#Payload List<String> payload) throws RuntimeException {
yourService.processIncomingData(payload);
}
And lastly, the properties:
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=helloworld
spring.kafka.listener.type=batch
spring.kafka.consumer.enable-auto-commit=false
# this is size of incoming list if broker has this many entries, can be lower eventually
spring.kafka.consumer.max-poll-records=100
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
The topology I am looking for is
So far I have not seen a way to define the topic per message in Cloud Stream. I understand that the consumers will be bound to specific topic but how does the producer sets the topic per message before sending the message to the exchange?
source.output().send(MessageBuilder.withPayload(myMessage).build());
Does not provide any way to set the topic for the exchange to route to the proper consumer.
Or maybe I don't understand something correctly?
UPDATE
I would expect not to receive the message in the consumer due to the bindingRoutingKey being 2222 and I am sending with routeTo 1111. But I still receive it on the consumer.
Producer Properties:
spring.rabbitmq.host=localhost
spring.rabbitmq.port=5672
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest
spring.cloud.stream.bindings.output.content-type=application/json
spring.cloud.stream.bindings.output.destination=messageExchange
spring.cloud.stream.rabbit.bindings.output.producer.routing-key-expression=headers['routeTo']
#EnableBinding(Source.class)
#SpringBootApplication
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}
Sender:
source.output().send(MessageBuilder.withPayload(mo).setHeader("routeTo", "1111").build());
And the Consumer:
spring.rabbitmq.host=localhost
spring.rabbitmq.port=5672
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest
spring.cloud.stream.bindings.input.destination=messageExchange
spring.cloud.stream.rabbit.bindings.input.consumer.bindingRoutingKey=2222
Application:
#SpringBootApplication
#EnableBinding(Sink.class)
public class Application {
private static final Logger log = LoggerFactory.getLogger(Application.class);
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
#StreamListener(Sink.INPUT)
public void ReceiveMo(String moDTO) {
log.info("Message received moDTO: {}", moDTO);
}
}
SECOND UPDATE
With the suggestions in the accepted answer below. I was able to make it work. Needed to remove the exchanges and queues from RabbitMQ using its UI and restart the RabbitMQ docker image.
The the routingKeyExpression rabbitmq producer property.
e.g. ...producer.routing-key-expression=headers['routeTo']
then
source.output().send(MessageBuilder.withPayload(myMessage)
.setHeader("routeTo", "Booking.new")
.build());
Note that the destination is the exchange name. By default, the binder expects a Topic exchange. If you wish to use a Direct exchange instead, you must set the exchangeType property.
I would like to use the new annotations and features provided in Spring 4.1 for an application that needs a JMS listener.
I've carefully read the notes in the Spring 4.1 JMS improvements post but I continue to miss the relationship between #JmsListener and maybe the DestinationResolver and how I would setup the application to indicate the proper Destination or Endpoint.
Here is the suggested use of #JmsListener
#Component
public class MyService {
#JmsListener(containerFactory = "myContainerFactory", destination = "myQueue")
public void processOrder(String data) { ... }
}
Now, I can't use this in my actual code because the "myQueue" needs to be read from a configuration file using Environment.getProperty().
I can setup an appropriate myContainerFactory with a DestinationResolver but mostly, it seems you would just use DynamicDestinationResolver if you don't need JNDI to lookup a queue in an app server and didn't need to do some custom reply logic. I'm simply trying to understand how Spring wants me to indicate the name of the queue in a parameterized fashion using the #JmsListener annotation.
Further down the blog post, I find a reference to this Configurer:
#Configuration
#EnableJms
public class AppConfig implements JmsListenerConfigurer {
#Override
public void configureJmsListeners(JmsListenerEndpointRegistrar registrar) {
registrar.setDefaultContainerFactory(defaultContainerFactory());
SimpleJmsListenerEndpoint endpoint = new SimpleJmsListenerEndpoint();
endpoint.setDestination("anotherQueue");
endpoint.setMessageListener(message -> {
// processing
});
registrar.registerEndpoint(endpoint);
}
Now, this makes some amount of sense and I could see where this would allow me to set a Destination at runtime from some external string, but this seems to be in conflict with using #JmsListener as it appears to be overriding the annotation in favor of endpoint.setMessageListener in the code above.
Any tips on how to specify the appropriate queue name using #JmsListener?
Also note that depending on use case you can already parameterize using properties file per environment and PropertySourcesPlaceholderConfigurer
#JmsListener(destinations = "${some.key}")
As per https://jira.spring.io/browse/SPR-12289
In case people are using #JmsListener with spring boot, you do not have to configure PropertySourcesPlaceholderConfigurer. It work's out the box
Sample:
class
#JmsListener(destination = "${spring.activemq.queue.name}")
public void receiveEntityMessage(final TextMessage message) {
// process stuff
}
}
application.properties
spring.activemq.queue.name=some.weird.queue.name.that.does.not.exist
Spring boot output
[26-Aug;15:07:53.475]-[INFO ]-[,]-[DefaultMes]-[o.s.j.l.DefaultMessageListenerContainer ]-[931 ]-Successfully refreshed JMS Connection
[26-Aug;15:07:58.589]-[WARN ]-[,]-[DefaultMes]-[o.s.j.l.DefaultMessageListenerContainer ]-[880 ]-Setup of JMS message listener invoker failed for destination 'some.weird.queue.name.that.does.not.exist' - trying to recover. Cause: User user is not authorized to read from some.weird.queue.name.that.does.not.exist
[26-Aug;15:07:59.787]-[INFO ]-[,]-[DefaultMes]-[o.s.j.l.DefaultMessageListenerContainer ]-[931 ]-Successfully refreshed JMS Connection
[26-Aug;15:08:04.881]-[WARN ]-[,]-[DefaultMes]-[o.s.j.l.DefaultMessageListenerContainer ]-[880 ]-Setup of JMS message listener invoker failed for destination 'some.weird.queue.name.that.does.not.exist' - trying to recover. Cause: User user is not authorized to read from some.weird.queue.name.that.does.not.exist
This proves that #JmsListener is able to pickup property values from application.properties without actually setting up any explicit PropertySourcesPlaceholderConfigurer
I hope this helps!
You could eventually do that right now but it's a bit convoluted. You can set a custom JmsListenerEndpointRegistry using JmsListenerConfigurer
#Configuration
#EnableJms
public class AppConfig implements JmsListenerConfigurer {
#Override
public void configureJmsListeners(JmsListenerEndpointRegistrar registrar) {
registrar.setEndpointRegistry(customRegistry());
}
}
and then override the registerListenerContainer method, something like
public void registerListenerContainer(JmsListenerEndpoint endpoint, JmsListenerContainerFactory<?> factory) {
// resolve destination according to whatever -> resolvedDestination
((AbstractJmsListenerEndpoint)endpoint).setDestination(resolvedDestination);
super.registerListenerContainer(endpoint, factory);
}
But we could do better. Please watch/vote for SPR-12280