I'm using Spring boot and specifically spring-kafka libraries for consuming from kafka.
I've a Class with #KafkaListener annotation on one of the methods to consume messages.I'm able to read in the messages. I realise its a topic and messages keep coming in based on the producer.
I want some kind of a cue that I'm at the last offset so I can trigger my re-conciliation logic with another source.How can I get hold of this cue.Is there a event that I can subscribe/listen to ?
You can set the container idleEventInterval property and (each consumer thread) will emit a ListenerContainerIdleEvent when no records have been received during that interval.
Use an ApplicationListener or #EventListener method to consume the event(s).
Related
I am new to Spring Boot #kafkaListener. My application receiving almost 200K message per second on topic. I want to separate message listener and processing of the message.
How can I use java.util.concurrent.BlockingQueue with #kafkaListener? Can I use it by using CompletableFuture?
Any sample code will help more.
I believe you want to have your consumer with pipelining implemented. Its not uncommon for one to implement this in a scenario like yours. Why? Well, the KafkaConsumer lacks in that decompressing / deserializing can be time consuming without considering the time it takes to do processing. Since these operations are stacked behind one thread, it would be ideal to separate the polling from the processing, which is achieved through a couple of buffers.
One way to do this: your EventReceiver spins up a thread for the polling. That thread would do the same thing you always do, but instead of firing off the listeners for each event, you'd pass the event to a receivedEvents buffer which could be BlockingQueue<RecieveEvent>. So in the for loop, you pass each record to the blocking queue. This thread would leverage another buffer once the for loop is over, like Queue<Map<TopicPartition, OffsetAndMetadata>> -- and it would commit the offsets that the processingThread has successfully processed.
Next, your EventReceiver spins up another thread - processingThread. This would handle pulling records from the buffer, firing the event to all the listeners for this receiver, and then update the Queues state for the pollingThread to commit.
Why doesn't the processingThread just commit the events instead of passing it back to the pollingThread? This is bc KafkaConsumer requires that the same thread that calls .poll() should be the one that calls consumer.commitAsync(...) or else you'll get a concurrency exception.
This approach doesn't work with auto commit enabled.
In terms of how one can do this using Spring Kafka, I'm not completely sure. However, I do know Spring Kafka separates EventReceiver from EventListener (#KafkaListener) which is separating the low-level kafka work from the business logic. In theory, you'd have to tune their implementation, but I think implementing this one without Spring Kafka library would be easier.
It is my first post to this here and I am not sure if this was covered here before, but here goes: I have a Kafka Streams application, using Processor API, following the topology below:
1. Consume data from an input topic (processor.addSource())
2. Inserts data into a DB (processor.addProcessor())
3. Produce its process status to an output topic (processor.addSink())
App works big time, however, for traceability purposes, I need to have in the logs the moment kstreams produced a message to the output topic, as well as its RecordMetaData (topic, partition, offset).
Example below:
KEY="MY_KEY" OUTPUT_TOPIC="MY-OUTPUT-TOPIC" PARTITION="1" OFFSET="1000" STATUS="SUCCESS"
I am not sure if there is a way to override the default kafka streams producer to add this logging or maybe creating my own producer to plug it on the addSink process. I partially achieved it by implementing my own ExceptionHandler (default.producer.exception.handler), but it only covers the exceptions.
Thanks in advance,
Guilherme
If you configure the streams application to use a ProducerInterceptor, then you should be able to get the information you need. Specifically, implementing the onAcknowledgement() will provide access to everything you listed above.
To configure interceptors in a streams application:
Properties props = new Properties();
// add this configuration in addition to your other streams configs
props.put(StreamsConfig.producerPrefix(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG), Collections.singletonList(MyProducerInterceptor.class));
You can provide more than one interceptor if desired, just add the class name and change the list implementation from a singleton to a regular List. Execution of the interceptors follows the order of the classes in the list.
EDIT: Just to be clear, you can override the provided Producer in Kafka Streams via the KafkaClientSupplier interface, but IMHO using an interceptor is the cleaner approach. But which direction to go is up to you. You pass in your KafkaClientSupplier in an overloaded Kafka Streams constructor.
I can't find this information anywhere. I have two queues, #JmsListener(destination = "p1"), #JmsListener(destination = "p2"). How can I make sure I only process 1 message at a time, even though I am listening to 2 queues, and also how do I configure the polling of what queue I get messages from first, that is after processing a message I want to poll p1 first. Or do weighted polling: p1:90%, p2:10%. Etc.
Basically I am asking how to implement priority processing of messages for Spring. I'm using SQS which doesn't support priorities.
Use one of the JmsTemplate receive() or receiveAndConvert() methods instead of the message-driven model.
Use transactions if you want to ensure no message loss.
I have a basic JMS related question in spring.
Rather than having to consume a single message at a time, it would be convenient to batch messages for a short duration (say a few seconds) and process them in bulk (thereby doing things in bulk). I see that java only provides an onMessage call that gives a single message at a time. I came across BatchMessageListenerContainer which seems to do this exactly. The recipe is ported to spring-batch where it is being used.
I wanted to know if there are any fundamental problem in the approach itself? If there are no problems, we can propose to the spring folks to add this in the spring-jms artifact itself (without needing to resort to use spring-batch whatsoever).
Thanks!
If your need is to process the messages in parallel you can use DefaultMessageListenerContainer in your spring project without the necessity for spring batch. You set the attribute concurrent consumers to the number of partitions you want.
#Bean
public DefaultMessageListenerContainer messageListener() {
DefaultMessageListenerContainer listener = new DefaultMessageListenerContainer();
**listener.setConcurrentConsumers(Integer.valueOf(env.getProperty(JmsConstant.CONCURRENT_CONSUMERS_SIZE)));**
// listener.setMaxConcurrentConsumers(maxConcurrentConsumers);
listener.setConnectionFactory((ConnectionFactory) queueConnectionFactory().getObject());
listener.setDestination((Destination) jmsQueue().getObject());
listener.setMessageListener(this);
listener.setSessionAcknowledgeMode(Session.AUTO_ACKNOWLEDGE);
listener.setSessionTransacted(true);
return listener;
}
Otherwise, if you're using spring batch, you can use remote chunking and BatchMessageListenerContainer, you can find an example here https://github.com/spring-projects/spring-batch/tree/master/spring-batch-samples/src/main/java/org/springframework/batch/sample/remotechunking
I have one doubt during my work with JMS. As I know it's possible to create synchrous message consumer. However, I must launch it with a frequency, because of the fact that there is no listener. Next, to consume messages synchrously from a queue I can create a MDB and set the pool to 1. I think it is not a good solution.
My aim is to consume messages synchrously when they appear on the queue. From my point of view above mentioned solutions are not good:
1. Consumer which is launched from time to time.
2. MDB (asynchrous normally) and pool is set to 1.
Are there any solutions for my purpose?
Not sure why you don't like MDBs... but if you want to avoid them, you could use the Spring JMS listener:
http://docs.spring.io/spring-framework/docs/current/spring-framework-reference/html/jms.html