I have jms message endpoint like:
#Bean
public JmsMessageDrivenEndpoint fsJmsMessageDrivenEndpoint(ConnectionFactory fsConnectionFactory,
Destination fsInboundDestination,
MessageConverter fsMessageConverter) {
return Jms.messageDrivenChannelAdapter(fsConnectionFactory)
.destination(fsInboundDestination)
.jmsMessageConverter(fsMessageConverter)
.outputChannel("fsChannelRouter.input")
.errorChannel("fsErrorChannel.input")
.get();
}
So, my questions is did I get next message before current message will be processed? If it will...Did it will get all messages in mq queue until it fills up all the memory? How to avoid it?
The JmsMessageDrivenEndpoint is based on the JmsMessageListenerContainer, its threading model and MessageListener callback for pulled messages. As long as your MessageListener blocks, it doesn't go to the next message in the queue to pull. When we build an integration flow starting with JmsMessageDrivenEndpoint, it becomes as a MessageListener callback. As long as we process the message downstream in the same thread (DirectChannel by default in between endpoints), we don't pull the next message from JMS queue. If you place a QueueChannel or an ExecutorChannel in between, you shift a processing to a different thread. The current one (JMS listener) gets a control back and it is ready to pull the next message. And in this case your concern about the memory is correct. You can still use QueueChannel with limited size or your ExecutorChannel can be configured with limited thread pool.
In any way my recommendation do not do any thread shifting in the flow when you start from JMS listener container. It is better to block for the next message and let the current transaction to finish its job. So you won't lose a message when something crashes.
Related
I am facing a scenario where the reply queue I connect to, runs out of handles. I have traced it to the fact that my JMS Producers are being cached but not my JMS consumers. I am able to send and receive messages just fine so there is no problem with connecting-sending-receiving to/from the queues. I am using the CachedConnectionFactory (SessionCacheSize = 10)with the target factory as com.ibm.mq.jms.MQQueueConnectionFactory while instantiating the jmsTemplate. Code snippet is as follows
:
:
String replyQueue = "MyQueue";// replyQueue which runs out of handles
messageCreator.setReplyToQueue(new MQQueue(replyQueue));
jmsTemplate.setReceiveTimeout(receiveTimeout);
jmsTemplate.send(destination, messageCreator);// Send to destination queue
Message message = jmsTemplate.receiveSelected(replyQueue,
String.format("JMSCorrelationID = '%s'", messageCreator.getMessageId()));
:
:
From the logs (jms TRACE is enabled) Producer is cached, so the destination queue "handle count" does not increase.
// The first time around (for Producer)
Registering cached JMS MessageProducer for destination[queue:///<destination>:com.ibm.mq.jms.MQQueueSender#c9x758b
// Second time around, the cached producer is reused
Found cached JMS MessageProducer for destination [queue:///<destination>]: com.ibm.mq.jms.MQQueueSender#c9x758b
However, the handles for the replyQueue keep increasing because for every call to that queue, I see a new JMS Consumer being registered. Ultimately the calls to open the replyQueue fail because of MQRC_HANDLE_NOT_AVAILABLE
// First time around
Registering cached JMS MessageConsumer for destination [queue:///<replyQueue>]:com.ibm.mq.jms.MQQueueReceiver#b3ytd25b
// Second time around, another MessageConsumer is registered !
Registering cached JMS MessageConsumer for destination [queue:///<replyQueue>]:com.ibm.mq.jms.MQQueueReceiver#re25b
My memory is a bit dim on this, but here is what is happening. You are receiving messages based on a message selector. This selector is always changing, however. As a test, either remove the selector or make it a constant and see what happens. So when you try to cache/pool based on connection/session/consumer, the consumer is always changing. This requires a new cache entry.
After you go through your 10 sessions, a new connection will be created, but the existing one is not closed. Increase your session count to 100, for example, and your connection count on the MQ broker should climb 10 time slower.
You need to create a new consumer for every message receive as your correlation ID is always changing. So just cache connection/session. No matter what you do, you will always have to round trip to the broker to ask for the new correlation ID.
I am trying to consume multiple message from a topic with manual ack but ack work if all message only by ack one time.
#KafkaListener(
id = "${kafka.buyers.product-sales-pricing.id}",
topics = "${kafka.buyers.product-sales-pricing.topic}",
groupId = "${kafka.buyers.group-id}",
concurrency = "${kafka.buyers.concurrency}"
)
public void listen( List<String> message, Acknowledgment ack ){}
In above code i am getting 5 message per consume if i put following configuration in spring boot property file:
kafka:
max-poll-records: 5 # Maximum number of records returned in a single call to poll()
but if i ack that listen then it ack all 5 message at same time.
Actually i want to ack separately for each message(means 5 message with 5 ack).
How can i do this in spring boot project?
When using a batch listener, the entire batch is acked when Acknowledgment.acknowledge() is called.
I would recommend using a single record listener rather than a batch listener for this use case.
listen(String msg, Acknowledgment ack)
It's not clear why you would commit offsets for only part of the batch.
If you must use a batch listener, it can still be done, but rather more complicated - you would need to get List<ConsumerRecord<?, ?>> to get topic/partition/offset information and also add Consumer<?, ?> consumer to the method parameters (and remove the Acknowledgment; you can then call commitOffsets() on the consumer however you want. But you MUST call it on the listener thread - the consumer is not thread-safe.
In my Spring Boot/Kafka project I have the following listener:
#KafkaListener(topics = "${kafka.topic.update}", containerFactory = "updateKafkaListenerContainerFactory")
public void onUpdateReceived(ConsumerRecord<String, Update> consumerRecord, Acknowledgment ack) {
// do some logic
ack.acknowledge();
}
Inside of the listener I need to check some condition according to my business logic and if it is not met - skip processing of this certain message and let Kafka know to redeliver this message one more time.
The reason I need this - according to the business logic of my application I need to avoid sending more than one post per second into the particular Telegram chat. This why I'd like to check the chatLastSent time in the Kafka listener and postpone message sending if needed(via message redelivery to this Kafka topic)
How to properly do it? Do I only need to not perform the ack.acknowledge(); this time or there is another, more proper way in order to achieve it?
Use the SeekToCurrentErrorHandler.
When you throw an exception, the container will invoke the error handler which will re-seek the unprocessed messages so they will be fetched again on the next poll.
You can use a RecordFilterStrategy.
See doc here : https://docs.spring.io/spring-kafka/docs/2.0.5.RELEASE/reference/html/_reference.html#_filtering_messages
Two questions:
I have an #StreamListener reading from a RabbitMQ channel. I have a pool of 500 ThreadTaskExecutor instances to process the messages as they are read.
The problem is that #StreamListener is reading messages even if the pool is completely utilized.
Caused by: org.springframework.core.task.TaskRejectedException:
Executor [java.util.concurrent.ThreadPoolExecutor#4c15ce96
[Running, pool size = 500, active threads = 500, queued tasks = 1500,
completed tasks = 1025020]] did not accept task:
org.springframework.cloud.sleuth.instrument.async.SpanContinuingTraceCallable#4dc03919
Is there a way to configure #StreamListener so that it only reads from the queue if it has capacity?
In addition, this error trickles up to an UndeclaredThrowableException. IO think its trying to throw the exception back to RabbitMQ so it reques the message. However the end is this:
[WARN] o.s.a.r.l.ConditionalRejectingErrorHandler
Execution of Rabbit message listener failed.
org.springframework.amqp.rabbit.listener.exception
.ListenerExecutionFailedException:
Retry Policy Exhausted
The final result is my message is lost.
Any suggestions for this second issue?
Did you try CallerRunsPolicy for your ThreadPoolTaskExecutor? This way the task won't finish with error and the thread from the SimpleMessageListenerContainer will be busy to do the latest task for just arrived message. As far as you don't use maxConcurrentConsumers option not new concurrent listeners will be raised and the current one (concurrentConsumers = 1 by default) will be busy and no new message is pulled from the Rabbit MQ.
See more info about listener container concurrency in the Docs. This way you may even reconsider your custom ThreadPoolTaskExecutor solution and will fully rely on the built-in mechanism in the Framework.
The maxConcurrency option is exposed for the RabbitMQ Binder Consumer as well.
In my application, I have a queue (HornetQ) set up on JBoss 7 AS.
I have used Spring batch to do some work once the messages is received (save values in database etc.) and then the consumer commits the JMS session.
Sometimes when there is an exception while processing the message, the excecution of consumer is aborted abruptly.
And the message remains in "in delivery" state. There are about 30 messages in this state on my production queue.
I have tried restarting the consumer but the state of these messages is not changed. The only way to remove these
messages from the queue is to restart the queue. But before doing that I want a way to read these messages so
that they can be corrected and sent to the queue again to be processed.
I have tried using QueueBrowser to read them but it does not work. I have searched a lot on Google but could not
find any way to read these messages.
I am using a Transacted session, where once the message is processed, I am calling:
session.commit();
This sends the acknowledgement.
I am implementing spring's
org.springframework.jms.listener.SessionAwareMessageListener
to recieve messages and then to process them.
While processing the messages, I am using spring batch to insert some data in database.
For a perticular case, it tries to insert data too big to be inserted in a column.
It throws an exception and transaction is aborted.
Now, I have fixed my producer and consumer not to have such data, so that this case should not happen again.
But my question is what about the 30 "in delivery" state messages that are in my production queue? I want to read them so that they can be corrected and sent to the queue again to be processed. Is there any way to read these messages? Once I know their content, I can restart the queue and submit them again (after correcting them).
Thanking you in anticipation,
Suvarna
It all depends on the Transaction mode you are using.
for instance if you use transactions:
// session here is a TX Session
MessageConsumer cons = session.createConsumer(someQueue);
session.start();
Message msg = consumer.receive...
session.rollback(); // this will make the messages to be redelivered
if you are using non TX:
// session here is auto-ack
MessageConsumer cons = session.createConsumer(someQueue);
session.start();
// this means the message is ACKed as we receive, doing autoACK
Message msg = consumer.receive...
//however the consumer here could have a buffer from the server...
// if you are not using the consumer any longer.. close it
consumer.close(); // this will release messages on the client buffer
Alternatively you could also set consumerWindowSize=0 on the connectionFactory.
This is on 2.2.5 but it never changed on following releases:
http://docs.jboss.org/hornetq/2.2.5.Final/user-manual/en/html/flow-control.html
I"m covering all the possibilities I could think of since you're not being specific on how you are consuming. If you provide me more detail then I will be able to tell you more:
You can indeed read your messages in the queue using jmx (with for example jconsole)
In Jboss As7 you can do it the following way :
MBeans>jboss.as>messaging>default>myJmsQueue>Operations
listMessagesAsJson
[edit]
Since 2.3.0 You have a dedicated method for this specific case :
listDeliveringMessages
See https://issues.jboss.org/browse/HORNETQ-763