Anyone know if a single listener can listens to multiple topic like below? I know just "topic1" works, what if I want to add additional topics? Can you please show example for both below? Thanks for the help!
#KafkaListener(topics = "topic1,topic2")
public void listen(ConsumerRecord<?, ?> record, Acknowledgment ack) {
System.out.println(record);
}
or
ContainerProperties containerProps = new ContainerProperties(new TopicPartitionInitialOffset("topic1, topic2", 0));
Yes, just follow the #KafkaListener JavaDocs:
/**
* The topics for this listener.
* The entries can be 'topic name', 'property-placeholder keys' or 'expressions'.
* Expression must be resolved to the topic name.
* Mutually exclusive with {#link #topicPattern()} and {#link #topicPartitions()}.
* #return the topic names or expressions (SpEL) to listen to.
*/
String[] topics() default {};
/**
* The topic pattern for this listener.
* The entries can be 'topic name', 'property-placeholder keys' or 'expressions'.
* Expression must be resolved to the topic pattern.
* Mutually exclusive with {#link #topics()} and {#link #topicPartitions()}.
* #return the topic pattern or expression (SpEL).
*/
String topicPattern() default "";
/**
* The topicPartitions for this listener.
* Mutually exclusive with {#link #topicPattern()} and {#link #topics()}.
* #return the topic names or expressions (SpEL) to listen to.
*/
TopicPartition[] topicPartitions() default {};
So, your use-case should be like:
#KafkaListener(topics = {"topic1" , "topic2"})
If we have to fetch multiple topics from the application.properties file :
#KafkaListener(topics = { "${spring.kafka.topic1}", "${spring.kafka.topic2}" })
Related
I have #KafkaListener:
#KafkaListener(topicPattern = "SameTopic")
public void onMessage(Message<String> message, Acknowledgment acknowledgment) {
String eventType = new String((byte[]) message.getHeaders().get("Event-Type"), StandardCharsets.UTF_8);
switch (eventType) {
case "create" -> doCreate(message);
case "update" -> doUpdate(message);
case "delete" -> doDelete(message);
}
}
Producer sets custom header Event-Type with three possible values: create, update, delete. Currently I'm reading this header value from Message and then invoke rest of the logic according to the header value.
Is there any way to create three #KafkaListeners where each of them will consume message filtered by some criteria - for my case filtered by header Event-Type value?
#KafkaListener(topicPattern = "SameTopic", ...)
public void onCreate(Message<String> message, Acknowledgment acknowledgment) {
doCreate(message);
}
#KafkaListener(topicPattern = "SameTopic", ...)
public void onUpdate(Message<String> message, Acknowledgment acknowledgment) {
doUpdate(message);
}
#KafkaListener(topicPattern = "SameTopic", ...)
public void onDelete(Message<String> message, Acknowledgment acknowledgment) {
doDelete(message);
}
I'm aware of RecordFilterStrategy, but couldn't get any help of it.
Consider to have those types mapped to the partition on the topic.
This way you definitely can have different #KafkaListener with the specific partition assigned:
/**
* The topicPartitions for this listener when using manual topic/partition
* assignment.
* <p>
* Mutually exclusive with {#link #topicPattern()} and {#link #topics()}.
* #return the topic names or expressions (SpEL) to listen to.
*/
TopicPartition[] topicPartitions() default {};
The doc is here: https://docs.spring.io/spring-kafka/docs/current/reference/html/#manual-assignment
It's probably not going to work well with several instances of your app, since with manual assignment there is no consumer group involved. You may consider to refine the logic to 3 different topics. Or if that is not possible from produce side, use Kafka Streams to split() the original topic to other topics according the record key.
I try to understand what's the type that returns when I aggregate in Spring Integration and that's pretty hard. I'm using Project Reactor and my code snippet is:
public FluxAggregatorMessageHandler randomIdsBatchAggregator() {
FluxAggregatorMessageHandler f = new FluxAggregatorMessageHandler();
f.setWindowTimespan(Duration.ofSeconds(5));
f.setCombineFunction(messageFlux -> messageFlux
.map(Message::getPayload)
.collectList()
.map(GenericMessage::new);
return f;
}
#Bean
public IntegrationFlow dataPipeline() {
return IntegrationFlows.from(somePublisher)
// ----> The type Message<?> passed? Or Flux<Message<?>>?
.handle(randomIdsBatchAggregator())
// ----> What type has been returned from the aggregation?
.handle(bla())
.get();
}
More than understanding the types that passes in the example, I want to know in general how can I know what are the objects that flows in the IntegrationFlow and their types.
IntegrationFlows.from(somePublisher)
This creates a FluxMessageChannel internally which subscribes to the provided Publsiher. Every single event is emitted from this channel to its subscriber - your aggregator.
The FluxAggregatorMessageHandler produces whatever is explained in the setCombineFunction() JavaDocs:
/**
* Configure a transformation {#link Function} to apply for a {#link Flux} window to emit.
* Requires a {#link Mono} result with a {#link Message} as value as a combination result
* of the incoming {#link Flux} for window.
* By default a {#link Flux} for window is fully wrapped into a message with headers copied
* from the first message in window. Such a {#link Flux} in the payload has to be subscribed
* and consumed downstream.
* #param combineFunction the {#link Function} to use for result windows transformation.
*/
public void setCombineFunction(Function<Flux<Message<?>>, Mono<Message<?>>> combineFunction) {
So, it is a Mono with a message which you really do with your .collectList(). That Mono is subscribed by the framework when it emits a reply message from the FluxAggregatorMessageHandler. Therefore your .handle(bla()) must expect a list of payloads. Which is really natural for the aggregator result.
See more in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#flux-aggregator
Actual SeekToCurrentErrorHandler has the ability to add not retryable exception, meaning all exception are retryable, except the initial one, and added X, Y, Z exceptions.
Stupid question : Is there a simple way to do the opposite : all exception are not retryable, except X', Y', Z'...
Yes, it is possible see this configuration method of that class:
/**
* Set an exception classifications to determine whether the exception should cause a retry
* (until exhaustion) or not. If not, we go straight to the recoverer. By default,
* the following exceptions will not be retried:
* <ul>
* <li>{#link DeserializationException}</li>
* <li>{#link MessageConversionException}</li>
* <li>{#link MethodArgumentResolutionException}</li>
* <li>{#link NoSuchMethodException}</li>
* <li>{#link ClassCastException}</li>
* </ul>
* All others will be retried.
* When calling this method, the defaults will not be applied.
* #param classifications the classifications.
* #param defaultValue whether or not to retry non-matching exceptions.
* #see BinaryExceptionClassifier#BinaryExceptionClassifier(Map, boolean)
* #see #addNotRetryableExceptions(Class...)
*/
public void setClassifications(Map<Class<? extends Throwable>, Boolean> classifications, boolean defaultValue) {
So, to make everything not-retryable you need to provide a default value as false.
The map should then container those exception you'd like to have retryable with the value for keys as true.
In Hadoop program, I tried to compress the result, I wrote the following code:
FileOutputFormat.setCompressOutput(job, true);
FileOutputFormat.setOutputCompressorClass(job, GzipCodec.class);
The result was compressed, and when I delete the first line:
FileOutputFormat.setCompressOutput(job, true);
and execute the program again, the result was same, was the above code
FileOutputFormat.setCompressOutput(job, true);
optional? What is the function of that code?
Please see the below methods in FileOutPutFormat.java which internally calls the method call which you have deleted.
i.e setCompressOutput(conf, true);
That means you are trying apply Gzip codec class then obviously its a pointer to code that output should be compressed. Isnt it ?
/**
* Set whether the output of the job is compressed.
* #param conf the {#link JobConf} to modify
* #param compress should the output of the job be compressed?
*/
public static void setCompressOutput(JobConf conf, boolean compress) {
conf.setBoolean("mapred.output.compress", compress);
}
/**
* Set the {#link CompressionCodec} to be used to compress job outputs.
* #param conf the {#link JobConf} to modify
* #param codecClass the {#link CompressionCodec} to be used to
* compress the job outputs
*/
public static void
setOutputCompressorClass(JobConf conf,
Class<? extends CompressionCodec> codecClass) {
setCompressOutput(conf, true);
conf.setClass("mapred.output.compression.codec", codecClass,
CompressionCodec.class);
}
After creating an Observable like so
var source = Rx.Observable.create(function(observer) {...});
What is the difference between subscribe
source.subscribe(function(x) {});
and forEach
source.forEach(function(x) {});
In the ES7 spec, which RxJS 5.0 follows (but RxJS 4.0 does not), the two are NOT the same.
subscribe
public subscribe(observerOrNext: Observer | Function, error: Function, complete: Function): Subscription
Observable.subscribe is where you will do most of your true Observable handling. It returns a subscription token, which you can use to cancel your subscription. This is important when you do not know the duration of the events/sequence you have subscribed to, or if you may need to stop listening before a known duration.
forEach
public forEach(next: Function, PromiseCtor?: PromiseConstructor): Promise
Observable.forEach returns a promise that will either resolve or reject when the Observable completes or errors. It is intended to clarify situations where you are processing an observable sequence of bounded/finite duration in a more 'synchronous' manner, such as collating all the incoming values and then presenting once, by handling the promise.
Effectively, you can act on each value, as well as error and completion events either way. So the most significant functional difference is the inability to cancel a promise.
I just review the latest code available, technically the code of foreach is actually calling subscribe in RxScala, RxJS, and RxJava. It doesn't seems a big different. They now have a return type allowing user to have an way for stopping a subscription or similar.
When I work on the RxJava earlier version, the subscribe has a subscription return, and forEach is just a void. Which you may see some different answer due to the changes.
/**
* Subscribes to the [[Observable]] and receives notifications for each element.
*
* Alias to `subscribe(T => Unit)`.
*
* $noDefaultScheduler
*
* #param onNext function to execute for each item.
* #throws java.lang.IllegalArgumentException if `onNext` is null
* #throws rx.exceptions.OnErrorNotImplementedException if the [[Observable]] tries to call `onError`
* #since 0.19
* #see ReactiveX operators documentation: Subscribe
*/
def foreach(onNext: T => Unit): Unit = {
asJavaObservable.subscribe(onNext)
}
def subscribe(onNext: T => Unit): Subscription = {
asJavaObservable.subscribe(scalaFunction1ProducingUnitToAction1(onNext))
}
/**
* Subscribes an o to the observable sequence.
* #param {Mixed} [oOrOnNext] The object that is to receive notifications or an action to invoke for each element in the observable sequence.
* #param {Function} [onError] Action to invoke upon exceptional termination of the observable sequence.
* #param {Function} [onCompleted] Action to invoke upon graceful termination of the observable sequence.
* #returns {Disposable} A disposable handling the subscriptions and unsubscriptions.
*/
observableProto.subscribe = observableProto.forEach = function (oOrOnNext, onError, onCompleted) {
return this._subscribe(typeof oOrOnNext === 'object' ?
oOrOnNext :
observerCreate(oOrOnNext, onError, onCompleted));
};
/**
* Subscribes to the {#link Observable} and receives notifications for each element.
* <p>
* Alias to {#link #subscribe(Action1)}
* <dl>
* <dt><b>Scheduler:</b></dt>
* <dd>{#code forEach} does not operate by default on a particular {#link Scheduler}.</dd>
* </dl>
*
* #param onNext
* {#link Action1} to execute for each item.
* #throws IllegalArgumentException
* if {#code onNext} is null
* #throws OnErrorNotImplementedException
* if the Observable calls {#code onError}
* #see ReactiveX operators documentation: Subscribe
*/
public final void forEach(final Action1<? super T> onNext) {
subscribe(onNext);
}
public final Disposable forEach(Consumer<? super T> onNext) {
return subscribe(onNext);
}