We are using Spring Cloud Reactive Streams with RabbitMQ.
Spring Reactive Stream appears to acknowledge the message as soon as it pulls it off the queue. So any errors unhandled exceptions that happens during the message processing need to be handled in the application (which is a different than a non-reactive stream where unhandled exceptions can be thrown and a message would be rejected, thus sending it to a dead letter queue).
How are we supposed to deal with a sudden shutdown in an application when a message is in flight?
For instance:
Application pulls message off of queue
Application marks message as acknowledged
Application starts message processing
Application is shut down before message processing is complete
When this happens, the message appears to be lost completely, since it is off of the queue but the application is stopped. How can we recover these messages?
You need to use manual acknowledgments and defer the acks until processing is asynchronously completed. To do that, you need to consume the whole message:
#Bean
public Consumer<Flux<Message<String>>> async() {
return inbound -> inbound
...
.map(msg -> {
try {
msg.getHeaders().get(AmqpHeaders.CHANNEL, Channel.class)
.basicAck(msg.getHeaders().get(AmqpHeaders.DELIVERY_TAG, Long.class), false);
}
catch (IOException e) {
e.printStackTrace();
}
return msg.getPayload();
})
.subscribe(System.out::println);
}
spring:
cloud:
stream:
function.definition: async
bindings:
async-in-0:
destination: testtock
group: async
rabbit:
bindings:
async-in-0:
consumer:
acknowledge-mode: MANUAL
prefetch: 10
Use basicReject to requeue or send to the DLQ.
Related
In my spring JMS listener onMessage method, I am using a reactive pipeline as as below where I receive a JMS message from a queue and transform that to a different format and publishing to another topic. Without the reactive flow, in the case of failure in output publishing, I can throw a runtimeexception for the same message to redeliver from the queue. But by using the below flow of reactive approach, I cannot find a way to throw an exception outside. Can you please help on what should be the approach here for throwing the exception outside to get the message redelivered
public void onMessage(Message message) {
Mono.just(message)
.doOnNext(LoggingUtil::logMessage)
.flatMapIterable(messageTransformer::transformToTriggerEvents)
.doOnNext(LoggingUtil::logTriggerEvent)
.doOnNext(triggerEventPublisher::publish)
.subscribe();
}
We have a service that send a message in a rabbit queue configure like this:
spring:
cloud:
stream:
bindings:
output:
destination: test
rabbit:
bindings:
output:
producer:
routingKeyExpression: headers.version
After having sent a very large number of messages during our bench tests, we noticed that the connection to the message queue is blocked:
Blocked queue
It seems that the blockage comes when the RabbitMQ server uses above 40% of the available RAM, it raises a memory alarm and blocks all connections that are publishing messages.
Our problem is that in this particular case, we lose the messages sent in the queue.
How to avoid loss of messages using spring cloud stream.
Here's a snippet of our class:
#Service
#EnableBinding(Source.class)
public class OurService {
Source source;
public void send(OurMessage ourMessage, Map<String, Object> headers) {
source.output().send(MessageBuilder.createMessage(ourMessage, new MessageHeaders(headers))));
}
}
This code does not return any exception even though the queue is blocked.
Is there a way to know the status of the queue (blocked) before sending the messages?
Add #EventListener methods (or ApplicationListener beans) to consume ConnectionBlockedEvent and ConnectionUnblockedEvent.
I have a spring boot application acting as a producer and sending messages via RabbitMQ.
In case the broker is down, is there some kind of listener that will detect when the broker is up again in order to retry sending the failed messages?
Thanks!
Not if you are only producing messages. You could add a #Scheduled method (and #EnableScheduling) that attempts to create a connection every so often and you can add a ConnectionListener to the connection factory which will be called when the new connection is opened.
Since the connection is shared, it won't hurt anything having this scheduled task running when the connection is already open...
#Scheduled(fixedDelay = 10000)
public void tryConn() {
try {
this.cachingConnectionFactory.createConnection().close();
}
catch (Exception e) {
}
}
On the consumer side, the listener container will keep retrying.
I'm using spring boot with spring-amqp and annotation based listener to consume message from a rabbitmq broker.
I've a spring component which contains a method like this:
#RabbitListener(queues = "tasks")
public void receiveMessage(#Payload Task task) {...}
I'm using the AUTO mode to acknowledge messages after successful execution of receiveMessage(...). If i detect a special error, i'm throwing AmqpRejectAndDontRequeueException to get this message into a configured dead letter queue. Now i need to nack a message only, so that the message gets requeued into the main queue of rabbitmq and another consumer has the possibility to work on that message again.
Which exception should i throw for that? I wouldn't like to use channel.basicNack(...) like described here (http://docs.spring.io/spring-integration/reference/html/amqp.html) if possible.
As long as defaultRequeueRejected is true (the default) in the container factory, throwing any exception other than AmqpRejectAndDontRequeueException will cause the message to be rejected and requeued.
The exception must not have a AmqpRejectAndDontRequeueException in its cause chain (the container traverses the causes to ensure there is no such exception).
I'm using direct exchange to publish messages to certain queues with routing keys, all configured in rabbit-server not code, I'm consuming messages with spring micro-service then some failures happens within the receiving method, then the message re-queued causing loop, so I'd like to add a policy with rabbit-server to prevent that kind of re-queuing, could it be added as an argument while binding queue with exchange with specific routing key, or it should be a policy ?
On any exception by default spring send nack with requeue "true". If in your spring consumer application you want to send requeue false, then throw the exception "AmqpRejectAndDontRequeueException". So your consumer code should loook something like this :
`void onMessage(){
try{
// Your Code Here
} catch(Exception e){
throw new AmqpRejectAndDontRequeueException();
}
}`