In my project, I am using Spring Kafka listener to consume messages from Kafka. I have a doubt that if the consume method code gets blocked due to some reason and never returned back, in this case, will this listener be able to receive new messages and proceed further or it will be hanged? In my case, it looks like, Kafka listener also got blocked and not processing further messages, even, another consumer of same group is also not receiving messages.
No; you will not get more records while a thread is blocked, unless the concurrency is > 1 and there are at least that many partitions. Even then, you will receive no more messages for the partition(s) assigned to the blocked consumer.
Related
I have Kafka listener writing data to a database, in case of database timeout (JdbcException), I'd like to retry, and if timeouts persist, stop consuming Kafka message.
As far as I understand, Spring Kafka 2.9 has 2 CommonErrorHandler implementations:
DefaultErrorHandler tries to redeliver messages several times and then send failed messages to logs or DLQ
CommonContainerStoppingErrorHandler stops listener container and message consumption
I would like to chain both of them: first try to redeliver messages several times, then stop container when delivery doesn't succeed.
How can I do that?
Use a DefaultErrorHandler with a custom recoverer that calls a CommonContainerStoppingErrorHandler after the retries are exhausted.
See this answer for an example
How do you exit spring boot application programmatically when retries are exhausted, to prevent kafka offset commit
(It uses the older SeekToCurrentErrorHandler, but the same concept applies.)
I need to know about the processing of consumed message(thread flow) via Spring SimpleMessageListenerContainer
I have following understanding
1) Messages consumed via consumer threads. (you can define the consumer thread pools via task executors).
2) the same consumer thread who receives the message process it and gets blocked until it does not finish the execution of handler method.
3) meanwhile other consumer threads gets created to consume the other messages and process those message. The interval to create those consumer threads is based on setStartConsumerMinInterval settings.
Please let me know if I am correct?
The next part is
I want to separate the consuming of message and processing of message in different threads(differnt pools for consuming and processing) how we can do that?
I have tried this way, I have made the handle message of handler as #Async to run it in different threads. Is it a correct way or any better way is available?
The last part is
in my Spring boot application I am both publishing and consuming messages, and I am using a single connection factory(CachingConnectionFactory). Should I use 2 connections factories 1 for publishing and other for consuming? and pass respecctive connection factory to the publishing and consuming beans?
For a Spring Framework app working with ActiveMQ and with/without WebSocket
The requirement is prior to send any message to a Topic a check should be done about Number Of Consumers, if it returns 1 the message can be sent safely. If it returns 0, the message can't be sent.
The clients could be come from websocket and consider there is no durable subscription. Thus if a message is sent, and there are no clients, the message arrives to the Topic and practically is lost (never consumed) and Messages Enqueued increments +1
I already did a research and I have read the following:
JMSTemplate check if topic exists and get subscriber count
Notify ActiveMQ producer if consumer on the destination is down
ActiveMQ get number of consumers listening to a topic from java
Detect change in Consumers of an ActiveMQ topic
Practically all is based on Advisory Message. I already have read:
Advisory Message
Handling Advisory Messages
Apache ActiveMQ Advisory Example
I understand that if exists a Topic named abc.xyz then ActiveMQ creates ActiveMQ.Advisory.Consumer.Topic.abc.xyz, until here I am ok with this pattern. I can confirm this approach through ActiveMQ Web Console
What is confuse for me is that practically all the examples available from the previous links works around creating a Session and uses mostly the onMessage method. For the latter I know it works how a listener.
Question 01: Thus who is expected to call that ActiveMQ.Advisory.Consumer.Topic.abc.xyz? It to trigger that onMessage method? That is my confusion.
What I need is work with the Spring Framework API (The app is already working and running with a CachingConnectionFactory, thus a Connection can be retrieved and other #Beans about infrastructure about ActiveMQ) and get access to that ActiveMQ.Advisory.Consumer.Topic.abc.xyz destination and retrieve the Number Of Consumers value.
Note: even when exists ActiveMQTopic declared with #Bean and is possible to retrieve that Destination for some #Component, sadly the API does not offer a method such getConsumers().
Question 02: How can be accomplished this?
I am assuming the JMS API for 2.0.x could help perhaps in someway.
I am using spring integration with weblogic jms. my logic is to put the json object in jms queue, and consume it from queue into the channel, validate it and route it based on a particular field. if there is any error, do the fix and put it back into the queue. I have two issues. 1. when the message is consumed into the object, I still can see the object as pending in administrative console of weblogic. 2. After fixing the validation, if put the modified object in the queue, I am getting the original object from queue.
<int-jms:outbound-channel-adapter id="jmsOutbound"
channel="requestChannel" connection-factory="queueConnectionFactory"
destination="inputQueue" />
<int-jms:message-driven-channel-adapter
id="jmsInbound" connection-factory="queueConnectionFactory"
destination="inputQueue" channel="routingChannel" />
if (message.getHeaders().get("documentType").equals("sec"))
routingChannels.add(outboundSecChannel);
else if (message.getHeaders().get("documentType").equals("unds"))
routingChannels.add(outboundFChannel);
else if (message.getHeaders().get("documentType").equals("CH"))
routingChannels.add(outboundAChannel);
else{
routingChannels.add(errorChannel);
}
putting in channel using routing
thanks for your help.
I think you should distinguish the consuming part from producing to the separate threads. I don't tell that you have to switch to the transactions, but at least the simple acknowledge for the consumed message should be done.
Since you tell that you are going to putt the message back to the queue, that's definitely the fact to always acknowledge the consumed message independently of the error fact. So, what I suggest is something like to place a QueueChannel or an ExecutorChannel somewhere after <int-jms:message-driven-channel-adapter> and before <int-jms:outbound-channel-adapter> to let them do their hard work with the WebLogic JMS in their own threads.
I am developing an application in which I have jms message queue.
There is a producer which enqueue message to the queue and a consumer to dequeue the message.
There might be cases when consumer is not running. If a message is not consumed by the consumer within a certain amount of time I need to catch that from producer.
I want to use JMX to monitor message queue's message whether it is expired.
Any suggestion or sample code how to this.....
It depends on the JMX implementer... Some servers provide JMX implementations to monitor its resources. If its not provided, then you will need to write the JMX implementation that uses the API provided by the MQ implementer.
An easier way to solve this problem is to use the request-response pattern with expiry. The consumer needs to respond in a specified internal of time. If it can't then the message on the queue can expire. If the response is not received the producer can take further action. JMS selector with correlation ID can be used to relate the responses with the request.