I have a basic JMS related question in spring.
Rather than having to consume a single message at a time, it would be convenient to batch messages for a short duration (say a few seconds) and process them in bulk (thereby doing things in bulk). I see that java only provides an onMessage call that gives a single message at a time. I came across BatchMessageListenerContainer which seems to do this exactly. The recipe is ported to spring-batch where it is being used.
I wanted to know if there are any fundamental problem in the approach itself? If there are no problems, we can propose to the spring folks to add this in the spring-jms artifact itself (without needing to resort to use spring-batch whatsoever).
Thanks!
If your need is to process the messages in parallel you can use DefaultMessageListenerContainer in your spring project without the necessity for spring batch. You set the attribute concurrent consumers to the number of partitions you want.
#Bean
public DefaultMessageListenerContainer messageListener() {
DefaultMessageListenerContainer listener = new DefaultMessageListenerContainer();
**listener.setConcurrentConsumers(Integer.valueOf(env.getProperty(JmsConstant.CONCURRENT_CONSUMERS_SIZE)));**
// listener.setMaxConcurrentConsumers(maxConcurrentConsumers);
listener.setConnectionFactory((ConnectionFactory) queueConnectionFactory().getObject());
listener.setDestination((Destination) jmsQueue().getObject());
listener.setMessageListener(this);
listener.setSessionAcknowledgeMode(Session.AUTO_ACKNOWLEDGE);
listener.setSessionTransacted(true);
return listener;
}
Otherwise, if you're using spring batch, you can use remote chunking and BatchMessageListenerContainer, you can find an example here https://github.com/spring-projects/spring-batch/tree/master/spring-batch-samples/src/main/java/org/springframework/batch/sample/remotechunking
Related
I want an advise according usage of spring cloud stream technologies.
Currently my service use spring-boot and implements some event-based approaches.
But the events are not sent to some kind of broker, but are simply handled by handlers in separate threads.
I am interested in spring cloud stream technology.
I have implemented CustomMessageRoutingCallback as shown in this example https://github.com/spring-cloud/spring-cloud-stream-samples/tree/main/routing-samples/message-routing-callback.
But the problem, that declaring all consumers at config in this way sounds like a pain:
#Bean
public Consumer<Menu> menuConsumer(){
return menu -> log.info(menu.toString());
}
Because I have around 50-60 different event types. Is where any way to register consumers dynamicly? Or the better way will be declare consumer with some raw input type, then deserialize message in consumer and manually route message to the right consumer?
This really has nothing to do with s-c-stream and more of an architectural question. If you have 50+ different event types having that many diff3rent consumers would be the least of your issues. The question I would be asking - is it really feasible to trust a single application to process that many different event types? What if a single event processing results in the system failure. Are you willing to live with non of the events being processed until the problem is fixed?
This is just an example, but there are many other architectural questions that would need to be answered before you can select a technology
A possible option is to create a common interface for your events
#Bean
public Consumer<CommonIntefaceType> menuConsumer(){
return commonIntefaceTypeObj -> commonIntefaceTypeObj.doSomething();
}
I am new to Spring Boot #kafkaListener. My application receiving almost 200K message per second on topic. I want to separate message listener and processing of the message.
How can I use java.util.concurrent.BlockingQueue with #kafkaListener? Can I use it by using CompletableFuture?
Any sample code will help more.
I believe you want to have your consumer with pipelining implemented. Its not uncommon for one to implement this in a scenario like yours. Why? Well, the KafkaConsumer lacks in that decompressing / deserializing can be time consuming without considering the time it takes to do processing. Since these operations are stacked behind one thread, it would be ideal to separate the polling from the processing, which is achieved through a couple of buffers.
One way to do this: your EventReceiver spins up a thread for the polling. That thread would do the same thing you always do, but instead of firing off the listeners for each event, you'd pass the event to a receivedEvents buffer which could be BlockingQueue<RecieveEvent>. So in the for loop, you pass each record to the blocking queue. This thread would leverage another buffer once the for loop is over, like Queue<Map<TopicPartition, OffsetAndMetadata>> -- and it would commit the offsets that the processingThread has successfully processed.
Next, your EventReceiver spins up another thread - processingThread. This would handle pulling records from the buffer, firing the event to all the listeners for this receiver, and then update the Queues state for the pollingThread to commit.
Why doesn't the processingThread just commit the events instead of passing it back to the pollingThread? This is bc KafkaConsumer requires that the same thread that calls .poll() should be the one that calls consumer.commitAsync(...) or else you'll get a concurrency exception.
This approach doesn't work with auto commit enabled.
In terms of how one can do this using Spring Kafka, I'm not completely sure. However, I do know Spring Kafka separates EventReceiver from EventListener (#KafkaListener) which is separating the low-level kafka work from the business logic. In theory, you'd have to tune their implementation, but I think implementing this one without Spring Kafka library would be easier.
It is my first post to this here and I am not sure if this was covered here before, but here goes: I have a Kafka Streams application, using Processor API, following the topology below:
1. Consume data from an input topic (processor.addSource())
2. Inserts data into a DB (processor.addProcessor())
3. Produce its process status to an output topic (processor.addSink())
App works big time, however, for traceability purposes, I need to have in the logs the moment kstreams produced a message to the output topic, as well as its RecordMetaData (topic, partition, offset).
Example below:
KEY="MY_KEY" OUTPUT_TOPIC="MY-OUTPUT-TOPIC" PARTITION="1" OFFSET="1000" STATUS="SUCCESS"
I am not sure if there is a way to override the default kafka streams producer to add this logging or maybe creating my own producer to plug it on the addSink process. I partially achieved it by implementing my own ExceptionHandler (default.producer.exception.handler), but it only covers the exceptions.
Thanks in advance,
Guilherme
If you configure the streams application to use a ProducerInterceptor, then you should be able to get the information you need. Specifically, implementing the onAcknowledgement() will provide access to everything you listed above.
To configure interceptors in a streams application:
Properties props = new Properties();
// add this configuration in addition to your other streams configs
props.put(StreamsConfig.producerPrefix(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG), Collections.singletonList(MyProducerInterceptor.class));
You can provide more than one interceptor if desired, just add the class name and change the list implementation from a singleton to a regular List. Execution of the interceptors follows the order of the classes in the list.
EDIT: Just to be clear, you can override the provided Producer in Kafka Streams via the KafkaClientSupplier interface, but IMHO using an interceptor is the cleaner approach. But which direction to go is up to you. You pass in your KafkaClientSupplier in an overloaded Kafka Streams constructor.
I'm using Spring boot and specifically spring-kafka libraries for consuming from kafka.
I've a Class with #KafkaListener annotation on one of the methods to consume messages.I'm able to read in the messages. I realise its a topic and messages keep coming in based on the producer.
I want some kind of a cue that I'm at the last offset so I can trigger my re-conciliation logic with another source.How can I get hold of this cue.Is there a event that I can subscribe/listen to ?
You can set the container idleEventInterval property and (each consumer thread) will emit a ListenerContainerIdleEvent when no records have been received during that interval.
Use an ApplicationListener or #EventListener method to consume the event(s).
I am using the Spring DefaultMessageListenerContainer to gain some dynamic benefits in setting the MessageSelector value since I am using the Glassfish OpenMQ which is not that advanced in that regards.
Let's have a JMS message. The listener issues a specific failure that means: retry after x seconds. It tries again with failure: retry after x*y seconds, and so on the time grows exponentially. If you cannot handle it after z retries, consider it as a poison JMS message.
DefaultMessageListenerContainer dmlc;
dmlc.stop();
dmlc.setMessageSelector(String.format("retries < %d AND retryTime <= %d", z, System.currentTimeMillis()));
dmlc.start();
I am not that satisfied with this solution, especially, when the Spring docs raise warning here:-). However, for the moment things meet our needs.
Now, I have a number of EJBs message consumers on different applications. Some of them need such dynamic changes of the messageSelector. Unfortunately, and to-my-best-knowledge, EJB MDBs do not support such dynamic "features". For example, see this.
Is that correct? is there a workaround for an EJB solution? I would appreciate any help.
To achieve dynamic changes to the message selector, you'd need to implement it straight in JMS, e.g.
ConnectionFactory cf;
Connection connection = cf.createConnection();
session = connection.createSession(transactional, acknowledgeMode);
MessageConsumer messageConsumer = session.createConsumer(destination, "message selector");
Additionally, you'd need to place this code some place it executes on its own, perhaps in an asynchronous task? But you'd be reinventing the wheel, as Spring DMLC does that better.
I don't know why you're doing this:
for load balancing? The message broker should take care of this.
for handling temporary downtimes? The queue should be configured to be able to store appropriate number of messages, or switch delivery to other node in cluster.