JMS Configuring backoff/retry without blocking onMessage() - spring-boot

javax.JMS version 2.0.1
Provider : ibm.mq v9.0
Framework : Java Spring boot
From what I know, onMessage() is asynchronous. I am successfully retrying the message send. However, the re-sending of messages happens instantaneously after a message failure. Ideally I want the retry to happen in a sliding window style eg. First retry after 20 seconds, second retry after 40 etc.
How can I achieve this without a Thread.Sleep() which, I presume, will block the entire Java thread and is not something I want at all ?
Code is something like this
final int TIME_TO_WAIT = 20;
public void onMessage(Message , message)
{
:
:
int t = message.getIntProperty("JMSXDeliveryCount");
if(t > 1)
{
// Figure out a way to wait for (TIME_TO_WAIT * t)
}
}
catch(Exception e)
{
// Do some logging/cleanup etc.
throw new RunimeException(e);// this causes a message retry
}

I would suggest you use exponential backoff in the retry logic, but you would need to use the Delivery Delay feature.
Define a custom JmsTemplate that will use delay property from the message, you should add retry count in the message property as well so that you can delay as per your need like 20, 40, 80, 160, etc
public class DelayedJmsTemplate extends JmsTemplate {
public static String DELAY_PROPERTY_NAME = "deliveryDelay";
#Override
protected void doSend(MessageProducer producer, Message message) throws JMSException {
long delay = -1;
if (message.propertyExists(DELAY_PROPERTY_NAME)) {
delay = message.getLongProperty(DELAY_PROPERTY_NAME);
}
if (delay >= 0) {
producer.setDeliveryDelay(delay);
}
if (isExplicitQosEnabled()) {
producer.send(message, getDeliveryMode(), getPriority(), getTimeToLive());
} else {
producer.send(message);
}
}
}
Define Components, that will have the capability fo re-enqueue of the message, you can define this interface in the base message listener. The handleException method should do all the tasks of enqueue and computing delay etc. You may not always interested in enqueuing, in some cases, you would discard messages as well.
You can see a similar post-processing logic here
https://github.com/sonus21/rqueue/blob/4c9c5c88f02e5cf0ac4b16129fe5b880411d7afc/rqueue-core/src/main/java/com/github/sonus21/rqueue/listener/PostProcessingHandler.java
#Component
#Sl4j
public class MessageListener {
private final JmsTemplate jmsTemplate;
#Autowired
public MessageListener(JmsTemplate jmsTemplate) {
this.jmsTemplate = jmsTemplate;
}
#JmsListener(destination = "myDestination")
public void onMessage(Message message) throws JMSException {
try {
// do something
} catch (Exception e) {
handleException("myDestination", message, e);
}
}
// Decide whether the message should be ignored due to many retries etc
private boolean shouldBeIgnored(String destination, Message message) {
return false;
}
// add logic to compute delay
private long getDelay(String destination, Message message, int deliveryCount) {
return 100L;
}
private void handleException(String destination, Message message, Exception e) throws JMSException {
if (shouldBeIgnored(destination, message)) {
log.info("destination: {}, message: {} is ignored ", destination, message, e);
return;
}
if (message.propertyExists("JMSXDeliveryCount")) {
int t = message.getIntProperty("JMSXDeliveryCount");
long delay = getDelay(destination, message, t + 1);
message.setLongProperty(DELAY_PROPERTY_NAME, delay);
message.setIntProperty("JMSXDeliveryCount", t + 1);
jmsTemplate.send(destination, session -> message);
} else {
// no delivery count, is this the first message or should be ignored?
}
}
}

Related

Springboot Kafka #Listener consumer pause/resume not working

I have a springboot Kafka Consumer & Producer. The consumer is expected to read data from topic 1 by 1, process(time consuming) it & write it to another topic and then manually commit the offset.
In order to avoid rebalancing, I have tried to call pause() and resume() on KafkaContainer but the consumer is always running & never responds to pause() call, tried it even with a while loop and faced no success(unable to pause the consumer). KafkaListenerEndpointRegistry is Autowired.
Springboot version = 2.6.9, spring-kafka version = 2.8.7
#KafkaListener(id = "c1", topics = "${app.topics.topic1}", containerFactory = "listenerContainerFactory1")
public void poll(ConsumerRecord<String, String> record, Acknowledgment ack) {
log.info("Received Message by consumer of topic1: " + value);
String result = process(record.value());
producer.sendMessage(result + " topic2");
log.info("Message sent from " + topicIn + " to " + topicOut);
ack.acknowledge();
log.info("Offset committed by consumer 1");
}
private String process(String value) {
try {
pauseConsumer();
// Perform time intensive network IO operations
resumeConsumer();
} catch (InterruptedException e) {
log.error(e.getMessage());
}
return value;
}
private void pauseConsumer() throws InterruptedException {
if (registry.getListenerContainer("c1").isRunning()) {
log.info("Attempting to pause consumer");
Objects.requireNonNull(registry.getListenerContainer("c1")).pause();
Thread.sleep(5000);
log.info("kafkalistener container state - " + registry.getListenerContainer("c1").isRunning());
}
}
private void resumeConsumer() throws InterruptedException {
if (registry.getListenerContainer("c1").isContainerPaused() || registry.getListenerContainer("c1").isPauseRequested()) {
log.info("Attempting to resume consumer");
Objects.requireNonNull(registry.getListenerContainer("c1")).resume();
Thread.sleep(5000);
log.info("kafkalistener container state - " + registry.getListenerContainer("c1").isRunning());
}
}
Am I missing something? Could someone please guide me with the right way of achieving the required behaviour?
You are running the process() method on the listener thread so pause/resume will not have any effect; the pause only takes place when the listener thread exits the listener method (and after it has processed all the records received by the previous poll).
The next version (2.9), due later this month, has a new property pauseImmediate, which causes the pause to take effect after the current record is processed.
You can try like this. This work for me
public class kafkaConsumer {
public void run(String topicName) {
try {
Consumer<String, String> consumer = new KafkaConsumer<>(config);
consumer.subscribe(Collections.singleton(topicName));
while (true) {
try {
ConsumerRecords<String, String> consumerRecords = consumer.poll(Duration.ofMillis(80000));
for (TopicPartition partition : consumerRecords.partitions()) {
List<ConsumerRecord<String, String>> partitionRecords = consumerRecords.records(partition);
for (ConsumerRecord<String, String> record : partitionRecords) {
kafkaEvent = record.value();
consumer.pause(consumer.assignment());
/** Implement Your Business Logic Here **/
Once your processing done
consumer.resume(consumer.assignment());
try {
consumer.commitSync();
} catch (CommitFailedException e) {
}
}
}
} catch (Exception e) {
continue;
}
}
} catch (Exception e) {
}
}

How to accept http requests after shutdown signal in Quarkus?

I tried this:
void onShutdown(#Observes final ShutdownEvent event) throws InterruptedException {
log.infof("ShutdownEvent received, waiting for %s seconds before shutting down", shutdownWaitSeconds);
TimeUnit.SECONDS.sleep(shutdownWaitSeconds);
log.info("Continue shutting down");
}
But after receiving ShutdownEvent Quarkus already responds with 503 to http requests. Looks like this could be done with ShutdownListener in preShutdown method. I have implemented this listener but it does not get called yet. How do I register ShutdownListener?
Use case here is OpenShift sending requests to terminating pod.
Option 1: Create Quarkus extension
Instructions are here. ShutdownController is my own class implementing ShutdownListener where I have a sleep in preShutdown method.
class ShutdownControllerProcessor {
#BuildStep
FeatureBuildItem feature() {
return new FeatureBuildItem("shutdown-controller");
}
#BuildStep
ShutdownListenerBuildItem shutdownListener() {
// Called at build time. Default constructor will be called at runtime.
// Getting MethodNotFoundException when calling default constructor here.
return new ShutdownListenerBuildItem(new ShutdownController(10));
}
}
Option 2: Modify ShutdownRecorder private static final field
New shutdown listener can be added using reflection. This is a bit ugly solution.
registerIfNeeded() need to be called after Quarkus startup, for example with timer 1 second after #PostConstruct.
#ApplicationScoped
public class ListenerRegisterer {
public void registerIfNeeded() {
try {
tryToRegister();
} catch (NoSuchFieldException | IllegalAccessException e) {
throw new IllegalStateException(e);
}
}
private void tryToRegister() throws NoSuchFieldException, IllegalAccessException {
final var field = ShutdownRecorder.class.getDeclaredField("shutdownListeners");
field.setAccessible(true);
final var listeners = (List<ShutdownListener>) field.get(null);
if (listeners != null && !listeners.toString().contains("ShutdownController")) {
listeners.add(new ShutdownController(10));
setFinalStatic(field, listeners);
}
}
private static void setFinalStatic(final Field field, final Object newValue) throws NoSuchFieldException, IllegalAccessException {
field.setAccessible(true);
final var modifiersField = Field.class.getDeclaredField("modifiers");
modifiersField.setAccessible(true);
modifiersField.setInt(field, field.getModifiers() & ~Modifier.FINAL);
field.set(null, newValue);
}
}

SpringAMQP - Retry/Resend messages dlx

I'm trying to use a retry mechanism using DLX.
So, basically I want to send an message for 3 times and than stop and keep this message stopped on dlx queue;
What I did:
Created WorkQueue bound to WorkExchange
Created RetryQueue bound to RetryExchange
WorkQueue -> set x-dead-letter-exchange to RetryExchange
RetryQueue -> set x-dead-letter-exchange to WorkExchange AND x-message-ttl to 300000 ms (5 minutes)
So, now when I send any message to WorkQueue and it fail.. this message goes to RetryQueue for 5min and than back to WorkQueue.. but it can keep failing and I would do like to stop it after 3 attemps ...
It is possible? Is possible set to RetryQueue try to 3 times and after stop?
thanks.
There is no way to do this in the broker alone.
You can add code to your listener - examine the x-death header to determine how many times the message has been retried and discard/log it (and/or send it to a third queue) in your listener when you want to give up.
EDIT
#SpringBootApplication
public class So59741067Application {
public static void main(String[] args) {
SpringApplication.run(So59741067Application.class, args);
}
#Bean
public Queue main() {
return QueueBuilder.durable("mainQueue")
.deadLetterExchange("")
.deadLetterRoutingKey("dlQueue")
.build();
}
#Bean
public Queue dlq() {
return QueueBuilder.durable("dlQueue")
.deadLetterExchange("")
.deadLetterRoutingKey("mainQueue")
.ttl(5_000)
.build();
}
#RabbitListener(queues = "mainQueue")
public void listen(String in,
#Header(name = "x-death", required = false) List<Map<String, ?>> xDeath) {
System.out.println(in + xDeath);
if (xDeath != null && (long) xDeath.get(0).get("count") > 2L) {
System.out.println("Given up on this one");
}
else {
throw new AmqpRejectAndDontRequeueException("test");
}
}
}

How to fix weird "channelMax" error rabbitmq rpc?

I created simple client and server. Client sends rpc requests:
RabbitTemplate template.convertSendAndReceive(...) ;
Server receive it, and answers back:
#RabbitListener(queues = "#{queue.getName()}")
public Object handler(#Payload String key)...
Then I make client send rpc requests asynchronously, simultaneously(which produces lot of concurrent rpc requests).
And unexpectedly receive an error:
org.springframework.amqp.AmqpResourceNotAvailableException: The channelMax limit is reached. Try later.
at org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:59)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createBareChannel(CachingConnectionFactory.java:1208)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.access$200(CachingConnectionFactory.java:1196)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.doCreateBareChannel(CachingConnectionFactory.java:599)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:582)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:552)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:534)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$1400(CachingConnectionFactory.java:99)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel
Rabbitmq client seems create too many channels. How to fix it?
And why my client create them so many?
Channels are cached so there should only be as many channels as there are actual RPC calls in process.
You may need to increase the channel max setting on the broker.
EDIT
If your RPC calls are long-lived, you can reduce the time the channel is used by using the AsyncRabbitTemplate with an explicit reply queue, and avoid using the direct reply-to feature.
See the documentation.
EDIT2
Here is an example using the AsyncRabbitTemplate; it sends 1000 messages on 100 threads (and the consumer has 100 threads).
The total number of channels used was 107 - 100 for the consumers and only 7 were used for sending.
#SpringBootApplication
public class So56126654Application {
public static void main(String[] args) {
SpringApplication.run(So56126654Application.class, args);
}
#RabbitListener(queues = "so56126654", concurrency = "100")
public String slowService(String in) throws InterruptedException {
Thread.sleep(5_000L);
return in.toUpperCase();
}
#Bean
public ApplicationRunner runner(AsyncRabbitTemplate asyncTemplate) {
ExecutorService exec = Executors.newFixedThreadPool(100);
return args -> {
System.out.println(asyncTemplate.convertSendAndReceive("foo").get());
for (int i = 0; i < 1000; i++) {
int n = i;
exec.execute(() -> {
RabbitConverterFuture<Object> future = asyncTemplate.convertSendAndReceive("foo" + n);
try {
System.out.println(future.get(10, TimeUnit.SECONDS));
}
catch (InterruptedException e) {
Thread.currentThread().interrupt();
e.printStackTrace();
}
catch (ExecutionException e) {
e.printStackTrace();
}
catch (TimeoutException e) {
e.printStackTrace();
}
});
}
};
}
#Bean
public AsyncRabbitTemplate asyncTemplate(ConnectionFactory connectionFactory) {
return new AsyncRabbitTemplate(connectionFactory, "", "so56126654", "so56126654-replies");
}
#Bean
public Queue queue() {
return new Queue("so56126654");
}
#Bean
public Queue reeplyQueue() {
return new Queue("so56126654-replies");
}
}

JMS Connection delivering messages sent to the queue while the connection was stopped

I am facing an issue with JMS Connection stop() and start(). A simple java program illustrating the same is:
public class Test {
static Connection producerConn = null;
static BufferedWriter consumerLog = null;
static BufferedWriter producerLog = null;
public static final void main(String[] args) throws Exception {
ConnectionFactory cf = new ActiveMQConnectionFactory("failover:(tcp://localhost:61616)");
producerConn = cf.createConnection();
producerLog = new BufferedWriter(new FileWriter("produced.log"));
consumerLog = new BufferedWriter(new FileWriter("consumed.log"));
new Thread(new Runnable() {
public void run() {
try {
producerConn.start();
Session session = producerConn.createSession(false, Session.AUTO_ACKNOWLEDGE);
Queue queue = session.createQueue("SampleQ1");
MessageProducer producer = session.createProducer(queue);
Random random = new Random();
byte[] messageBytes = new byte[1024];
for (int i = 0; i < 100; i++) {
random.nextBytes(messageBytes);
Message message = session.createObjectMessage(messageBytes);
producer.send(message);
Thread.currentThread().sleep(10);
producerLog.write(message.getJMSMessageID());
producerLog.newLine();
producerLog.flush();
}
System.out.println("Produced 100000 messages...");
producerLog.close();
}
catch (Exception e) {
e.printStackTrace();
}
}
}).start();
System.out.println("Started producer...");
new Thread(new Runnable() {
public void run() {
int count = 0;
try {
producerConn.start();
Session session = producerConn.createSession(false, Session.AUTO_ACKNOWLEDGE);
Queue queue = session.createQueue("SampleQ1");
MessageConsumer consumer = session.createConsumer(queue);
consumer.setMessageListener(new Test().new MyListener());
}
catch (Exception e) {
e.printStackTrace();
}
}
}).start();
System.out.println("Started consumer...");
}
private class MyListener implements MessageListener{
private int count = 0;
public void onMessage(Message message) {
try {
message.acknowledge();
System.out.println("count is " +count++);
if(count == 5){
producerConn.stop();
System.out.println("Sleeping Now for 5 seconds. . ." +System.currentTimeMillis());
Thread.currentThread().sleep(5000);
producerConn.start();
}
System.out.println("Waking up . . ." +System.currentTimeMillis());
consumerLog.write(message.getJMSMessageID());
consumerLog.newLine();
consumerLog.flush();
}
catch (Exception e) {
e.printStackTrace();
}
}
}
}
My idea is to simulate the connection stop() and start(). Therefore, in the consumer thread after calling stop(), I have placed a sleep of 5 seconds. However, in the mean time the producer thread continues its job of sending message to the queue.
I expected the test to just consume only the message delivered before the consumer calls stop() and after it calls start() again after waking up from the sleep. But what's happening here is, when consumer wakes up it reads all the messages from the server even those that were sent to the queue when consumer's message reception was stopped.
Am I doing anything wrong here ?
There is nothing wrong there, it's the correct behavior. In asynchronous messaging producer and consumer are loosely decoupled. A producer does not care whether a consumer is consuming messages or not. It keeps putting messages to a queue while the consumer may be down, or stopped consuming messages or actively consuming messages.
The connection.stop() method has no effect on producer. It affects only consumer, stop() method pauses the delivery of messages from JMS provider to a consumer while start() method starts/resumes message delivery.
Hope this helped.

Resources