KafkaStream tombstone message insertion approach - apache-kafka-streams

I have a stream of documents, that go through multiple processing steps. These steps are done in parallel. After each step completes, a message is sent to stage completion topic. After all the steps are done, the tracker sends a message to processing complete topic with the document Id.
I am using kafka streams (with spring cloud stream on top) in the tracker implement the above functionality.
Following is the sample code.
#StreamListener
#SendTo("processingComplete")
public KStream<String, String> onCompletion(#Input("stageCompletion")
KStream<String, String> stageCompletionStream) {
return stageCompletionStream
.filter(this::checkValidity)
.groupByKey(Serialized.with(Serdes.String(), Serdes.String()))
.reduce(this::aggregateStageCompletion,
Materialized.as("stage_completion_store"))
.toStream()
.filter((ignored, message) -> checkCompletion(message))
.map(this::publishCompletion);
}
After I publish completion message, I need to clean up the state store - stage_completion_store (which happens to be rocks db by default) of that document Id.
The suggested approach is to insert a tombstone message; to do so I have additionally implemented another stream to read processing complete topic and merge the same with stage completion stream.
Follow is the code using this approach.
#StreamListener
#SendTo("processingComplete")
public KStream<String, String> onCompletion(#Input("stageCompletion")
KStream<String, String>
stageCompletionStream,#Input("processingCompleteFeed") KStream<String,
String> processingCompletionStream){
return processingCompletionStream.merge(stageCompletionStream)
.filter(this::checkValidity)
.groupByKey(Serialized.with(Serdes.String(),Serdes.String()))
.reduce(this::aggregateStageCompletion,
Materialized.as("stage_completion_store"))
.toStream()
.filter((ignored,message)->checkCompletion(message))
.map(this::publishCompletion);
}
The aggregateStageCompletion inserts the tombstone(returns null) when the message is a processing completion message.
Is this a good way to do it - read a stream just to mark tombstone? or is there a better approach to achieve the same?

Related

Spring Cloud Stream Kafka Streams inbound KTable predictable internal state-store topic names

We're using Kafka Streams with Spring Cloud Stream Functions. We have the typical example application which joins user clicks kstream with user regions ktable.
We know we can force custom names for internal changelog or repartition topics by using appropiate methods that accept a name for materialized store when defining our topology:
#Bean
public BiFunction<KStream<String, Long>, KTable<String, String>, KStream<String, Long>> bifunctionktable() {
return (userClicksStream, userRegionsTable) -> userClicksStream
.leftJoin(userRegionsTable,
(clicks, region) -> new RegionWithClicks(region == null ? "UNKNOWN" : region, clicks),
Joined.with(Serdes.String(), Serdes.Long(), null, "bifunctionktable-leftjoin"))
.map((user, regionWithClicks) -> new KeyValue<>(regionWithClicks.getRegion(), regionWithClicks.getClicks()))
.groupByKey(Grouped.with(Serdes.String(), Serdes.Long()).withName("bifunctionktable-groupbykey"))
.reduce((firstClicks, secondClicks) -> firstClicks + secondClicks, Materialized.as("bifunctionktable-reduce"))
.toStream();
}
But for the input KTable we cannot change its state-store internal topic name and we always get this topic name: myapp-id-user-regions-STATE-STORE-0000000001-changelog
If we were fully creating our topology by code we do have builder.table(final String topic, final Materialized<K, V, KeyValueStore<Bytes, byte[]>> materialized) method, but using functions... Is there any way to customize the internal topic name for the input KTable in this case?
You can add a custom name for the incoming KTable by using the following property:
spring.cloud.stream.kafka.streams.bindings.bifunctionktable-in-1.consumer.materializedAs: <Your-custom-store-name>
This is documented in this section of the reference docs.

Using Spring Cloud Stream Kafka Streams with Avro input/output with nativeEncoding/decoding=false

We're testing the use of Kafka Streams via Spring Cloud Stream function support with Avro input/output records, but setting nativeEncoding=false and nativeDecoding=false in order to use a custom MessageConverter where we do the Avro conversion.
The default serdes are StringSerde for keys and ByteArraySerde for values.
Everything is ok when we only use a KStream to KStream function, for example:
#Bean
public Function<KStream<String, DataRecordAvro>, KStream<String, DataRecordAvro>> wordsCount() {
return input -> input
.flatMapValues(value -> Arrays.asList(value.getName().toString().toLowerCase().split("\\W+")))
.map((key, value) -> new KeyValue<>(value, value))
.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
.windowedBy(TimeWindows.of(Duration.ofSeconds(5)).grace(Duration.ofMillis(0)))
.count()
.toStream()
.map((key, value) -> new KeyValue<>(key.key(), new DataRecordAvro(key.key(), value)));
}
but when we try a little bit more complex example involving an input KTable like this:
#Bean
public BiFunction<KStream<String, DataRecordAvro>, KTable<String, DataRecordAvro>, KStream<String, DataRecordAvro>> userClicksRegionKTableAvro() {
return (userClicksStream, usersRegionKTable) -> userClicksStream
.leftJoin(usersRegionKTable,
(clicks, region) -> new RegionWithClicks(region == null ? "UNKNOWN" : region.getName().toString(), clicks.getCount()))
.map((user, regionWithClicks) -> new KeyValue<>(regionWithClicks.getRegion(), regionWithClicks.getClicks()))
.groupByKey(Grouped.with(Serdes.String(), Serdes.Long()))
.reduce(Long::sum)
.mapValues((key, value) -> new DataRecordAvro(key, value))
.toStream();
}
(The DataRecordAvro class only have two members: CharSequence name; Long count;)
When received the first record this exception is thrown:
ClassCastException invoking Processor. Do the Processor's input types match the deserialized types? Check the Serde setup and change the default Serdes in StreamConfig or provide correct Serdes via method parameters. Make sure the Processor can accept the deserialized input of type key: java.lang.String, and value: com.xxxx.kstreams.fixtures.avro.DataRecordAvro.
Note that although incorrect Serdes are a common cause of error, the cast exception might have another cause (in user code, for example). For example, if a processor wires in a store, but casts the generics incorrectly, a class cast exception could be raised during processing, but the cause would not be wrong Serdes.
The processor where the exception is thrown seems to be:
KSTREAM-LEFTJOIN-0000000011:
states: [user-regions-avro-STATE-STORE-0000000008]
We have no idea why it doesn't work in this case. Maybe the leftJoin operation persists information to an internal topic and there the useNativeEncoding/Decoding=false are not taken into account? But why the kstream->kstream example above does work? We thought the Avro conversion was only done at the start and end of the Topology, why this casting exception while using leftJoin?
Here is another example that works ok (without input Avro records, leaving consumer useNativeDecoding as default true):
#Bean
public BiFunction<KStream<String, Long>, KTable<String, String>, KStream<String, DataRecordAvro>> userClicksRegionKTable() {
return (userClicksStream, usersRegionKTable) -> userClicksStream
.leftJoin(usersRegionKTable,
(clicks, region) -> new RegionWithClicks(region == null ? "UNKNOWN" : region, clicks))
.map((user, regionWithClicks) -> new KeyValue<>(regionWithClicks.getRegion(), regionWithClicks.getClicks()))
.groupByKey(Grouped.with(Serdes.String(), Serdes.Long()))
.reduce(Long::sum)
.mapValues((key, value) -> new DataRecordAvro(key, value))
.toStream();
}
Please help!
For Kafka Streams binder in Spring Cloud Stream, we recommend using native decoding/encoding with Serdes unless you have strong reasoning for relying on the message conversion approach. What is the use case that forces you to go with message converters here? In practice, using message converters for serialization purposes in Kafka Streams applications in Spring Cloud Stream adds an extra layer in your topology and makes it deeper, thus the recommendation to use native decoding/encoding.
As you noted, for KTable, the binder always uses native decoding - at the moment, it is not possible to use message converters there. When you turn off useNativeDecoding on the KTable binding, the binder ignores it and simply uses the default byte serde. I suggest going with the default on the KTable binding and then adding the following bean in your application configuration.
#Bean
public Serde< DataRecordAvro> dataRecordAvroSerde() {
// return Serde
}
This way the binder will detect this bean and realize that the Serde type matches with the type from the function signature and then use it on those inputs.
If you have further issues on this app, feel free to share an MCRE. We can take further look then.

How to create multi output stream from single input stream with Spring Cloud Kafka stream binder?

I am trying to create multi output streams(depend on different time window) from single input stream.
interface AnalyticsBinding {
String PAGE_VIEWS_IN = "pvin";
String PAGE_VIEWS _COUNTS_OUT_Last_5_Minutes = "pvcout_last_5_minutes";
String PAGE_VIEWS _COUNTS_OUT_Last_30_Minutes = "pvcout_last_30_minutes";
#Input(PAGE_VIEWS_IN)
KStream<String, PageViewEvent> pageViewsIn();
#Output(PAGE_VIEWS_COUNTS_OUT_Last_5_Minutes)
KStream<String,Long> pageViewsCountOutLast5Minutes();
#Output(PAGE_VIEWS_COUNTS_OUT_Last_30_Minutes)
KStream<String,Long> pageViewsCountOutLast30Minutes();
}
#StreamListener
#SendTo({ AnalyticsBinding.PAGE_VIEWS_COUNTS_OUT_Last_5_Minutes })
public KStream<String, Long> processPageViewEventForLast5Mintues(
#Input(AnalyticsBinding.PAGE_VIEWS_IN)KStream<String, PageViewEvent> stream) {
// aggregate by Duration.ofMinutes(5)
}
#StreamListener
#SendTo({ AnalyticsBinding.PAGE_VIEWS_COUNTS_OUT_Last_30_Minutes })
public KStream<String, Long> processPageViewEventForLast30Mintues(
#Input(AnalyticsBinding.PAGE_VIEWS_IN)KStream<String, PageViewEvent> stream) {
// aggregate by Duration.ofMinutes(30)
}
When I start the application just one stream task would work, Is there a way to get both processPageViewEventForLast5Mintues and processPageViewEventForLast30Mintues work simultaneously
You are using the same input binding in both processors and that's why you are seeing only one as working. Add another input binding in the binding interface and set it's destination to the same topic. Also, change one of the StreamListener methods to use this new binding name.
With that said, if you are using the latest versions of Spring Cloud Stream, you should consider migrating to a functional model. For e.g. the following should work.
#Bean
public Function<KStream<String, PageViewEvent>, KStream<String, Long>> processPageViewEventForLast5Mintues() {
...
}
and
#Bean
public Function<KStream<String, PageViewEvent>, KStream<String, Long>> processPageViewEventForLast30Mintues() {
...
}
The binder automatically creates two distinct input bindings in this case.
You can set destinations on those bindings.
spring.cloud.stream.bindings.processPageViewEventForLast5Mintues-in-0.destination=<your Kafka topic>
spring.cloud.stream.bindings.processPageViewEventForLast30Mintues-in-0.destination=<your Kafka topic>

Spring Integration with DSL: Can File Outbound Channel Adapter create file after say 10 mins of interval

I have a requirement where my application should read messages from MQ and write using file outbound channel adapter. I want each of my output file should contain messages of every 10 mins of interval. Is there any default implementation exist, or any pointers to do so.
public #Bean IntegrationFlow defaultJmsFlow()
{
return IntegrationFlows.from(
//read JMS topic
Jms.messageDrivenChannelAdapter(this.connectionFactory).destination(this.config.getInputQueueName()).errorChannel(errorChannel()).configureListenerContainer(c ->
{
final DefaultMessageListenerContainer container = c.get();
container.setSessionTransacted(true);
container.setMaxMessagesPerTask(-1);
}).get())
.channel(messageProcessingChannel()).get();
}
public #Bean MessageChannel messageProcessingChannel()
{
return MessageChannels.queue().get();
}
public #Bean IntegrationFlow messageProcessingFlow() {
return IntegrationFlows.from(messageProcessingChannel())
.handle(Files.outboundAdapter(new File(config.getWorkingDir()))
.fileNameGenerator(fileNameGenerator())
.fileExistsMode(FileExistsMode.APPEND).appendNewLine(true))
.get();
}
First of all you could use something like a QueueChannel with the poller on the endpoint for the FileWritingMessageHandler with the fixedDelay for those 10 mins. However you should keep in mind that messages are going to be stored in the memory before poller does its work. So, once a crash of your application, the messages are lost.
On the other hand you can use a JmsDestinationPollingSource with similar poller configuration. This way, however, you need to configure it with the maxMessagesPerPoll(-1) to let it to pull as much messages from the MQ as possible during single polling task - once in 10 mins.
Another variant is possible with an aggregator and its groupTimeout option. This way you won't have an output message from the aggregator until 10 mins interval passes. However again: the store is in memory by default. I wouldn't introduce one more persistence storage just to satisfy a periodic requirement when we already have an MQ and we really can poll exactly that. Therefore I would go a JmsDestinationPollingSource variant.
UPDATE
Can you help me with how to set fixed delay in file outbound adapter.
Since you deal with the QueueChannel, you need to configure for the "fixed delay" a PollingConsumer endpoint. This one really belongs to the subscriber of that channel. Indeed it is a .handle(Files.outboundAdapter) part. Only what you are missing that Poller is an option of the endpoint, not a MessageHandler. Consider to use an overloaded handle()variant:
.handle(Files.outboundAdapter(new File(config.getWorkingDir()))
.fileNameGenerator(fileNameGenerator())
.fileExistsMode(FileExistsMode.APPEND).appendNewLine(true),
e -> e.poller(p -> p.fixedDelay(10000)))
Or a sample example for JMSDestinationPollingSource
#Bean
public IntegrationFlow jmsInboundFlow() {
return IntegrationFlows
.from(Jms.inboundAdapter(cachingConnectionFactory())
.destination("jmsInbound"),
e -> e.poller(p -> p.fixedDelay(10000)))
.<String, String>transform(String::toUpperCase)
.channel(jmsOutboundInboundReplyChannel())
.get();
}

KTable returns no data in Spring Boot application, however it can be queried

I have a Spring Boot application working with Kafka Streams. I have a KTable with some financial currency quotes which is created like this:
#Bean(name = "indicativeQuotes")
public KTable<String, Quote> quoteKTable(StreamsBuilder streamsBuilder) {
return streamsBuilder.table(quoteTopicName,
Materialized.<String,Quote,KeyValueStore<Bytes,byte[]>>as("quoteTable")
.withKeySerde(Serdes.String())
.withValueSerde(new JsonSerde<>(Quote.class)));
}
I #Autowire this bean in another component, and test it with the following code:
#Autowired
private KTable<String, Quote> indicativeQuotes;
#PostConstruct
private void postConstruct() {
doPrint();
}
public void doPrint() {
ReadOnlyKeyValueStore<String, Quote> store = streamsBuilderFactoryBean.getKafkaStreams().store("quoteTable", QueryableStoreTypes.keyValueStore());
store.all().forEachRemaining(keyValue -> log.info("Key: " + keyValue.key + " Value: " + keyValue.value));
indicativeQuotes.foreach((k,v) -> log.info(k));}
The code logs correct values when querying through store, but it outputs nothing in foreach(), as if like table was empty. I have also tried print() and other options - all output nothing without any exceptions.
I'm starting to think that I cant inject KTable beans like that, but Spring documentation on the topic of kafka streams is pretty scarce and I can't find good examples. Any help will be appreciated.
Update.
My use case is that I have a scheduled Quartz job which is supposed to write current state of KTable to a Kafka topic when triggered, like following:
#Override
protected void executeInternal(JobExecutionContext jobExecutionContext) throws JobExecutionException {
TriggerKey triggerKey = jobExecutionContext.getTrigger().getKey();
log.info("Job was triggered by: {}", triggerKey.getName());
indicativeQuotes.filter((key, value) -> key.equals(triggerKey.getName()))
.mapValues(quoteToCourseFixedMapper)
.toStream()
.peek((instrument, course)-> log.info("Sending courses for instrument: {}, {}", instrument, course))
.to(quoteEventTopicName);
}
But I think this code does not work because it is not a part of processing topology and I cannot just take data from Ktable on demand. I'm a bit puzzled here, of course I can query the data through store when event is triggered, but maybe there is a better pattern for such use case? Basically I'm interested if its possible to incorporate this triggered job events as a part of processing pipeline.
If you just want to publish the updates to another topic, turn the KTable to a KStream and use the to() function.
KTable ktable = ...;
KStream ksteram = ktable.toStream();
kstream.to("topic", Produces.with(keySerde, valueSerde))
The topic will contain the change log of that table.
BUT
Apparently because of some life cycle related concepts, you can't just inject (#autowire) KStream/KTable. You should keep your KafkaStreams related code kind of as in-line as possible.
So in your specific case that you want the to do something with current state of table in some "random" time, you have to query the store (table). So search for kafka steams interactive queries. remember that you need to fetch data from all of your instances of the application (if you have more than 1. Or you can use a global store. Its a day or two of search.

Resources