While reading up on the suppress() documentation, I saw that the time window will not advance unless records are being published to the topic, because it's based on event time. Right now, my code is outputting the final value for each key, because traffic on the topic is constant, but there are downtimes when that system is brought down, causing existing records in the state store to be "frozen". I was wondering what the difference is between just having reduce(), instead of reduce().suppress(). Does reduce() act like suppress() in that they are both event time driven? My understanding is that both are doing the same thing, aggregating the keys within a certain time window.
My topology is the following:
final Map<String, String> serdeConfig = Collections.singletonMap("schema.registry.url", schemaRegistryUrl);
final Serde<EligibilityKey> keySpecificAvroSerde = new SpecificAvroSerde<EligibilityKey>();
keySpecificAvroSerde.configure(serdeConfig, true);
final Serde<Eligibility> valueSpecificAvroSerde = new SpecificAvroSerde<Eligibility>();
valueSpecificAvroSerde.configure(serdeConfig, false);
// KStream<EligibilityKey, Eligibility>
KStream<EligibilityKey, Eligibility> kStreamInput = builder.stream(input,
Consumed.with(keySpecificAvroSerde, valueSpecificAvroSerde));
// KStream<EligibilityKey, String>
KStream<EligibilityKey, String> kStreamMapValues = kStreamInput
.mapValues((key, value) -> Processor.process(key, value));
// WindowBytesStoreSupplier
WindowBytesStoreSupplier windowBytesStoreSupplier = Stores.inMemoryWindowStore("in-mem",
Duration.ofSeconds(retentionPeriod), Duration.ofSeconds(windowSize), false);
// Materialized
Materialized<EligibilityKey, String, WindowStore<Bytes, byte[]>> materialized = Materialized
.as(windowBytesStoreSupplier);
materialized = Materialized.with(keySpecificAvroSerde, Serdes.String());
// TimeWindows
TimeWindows timeWindows = TimeWindows.of(Duration.ofSeconds(size)).advanceBy(Duration.ofSeconds(advance))
.grace(Duration.ofSeconds(afterWindowEnd));
// KTable<Windowed<EligibilityKey>, String>
KTable<Windowed<EligibilityKey>, String> kTable = kStreamMapValues
.groupByKey(Grouped.with(keySpecificAvroSerde, Serdes.String())).windowedBy(timeWindows)
.reduce((a, b) -> b, materialized.withLoggingDisabled().withRetention(Duration.ofSeconds(retention)))
.suppress(Suppressed.untilWindowCloses(Suppressed.BufferConfig.unbounded().withLoggingDisabled()));
// KStream<Windowed<EligibilityKey>, String>
KStream<Windowed<EligibilityKey>, String> kStreamOutput = kTable.toStream();
By using reduce() without suppress, the result of the aggregation is updated continuously, i.e., updates to the KTable that holds the results of the reduce() are sent downstream also before all records of a window are processed.
Assume a reduce that just sums up the values in a window of duration 3 with grace 0 and the following input records (key, value, timestamp) to reduce():
input record (A, 1, 1) of W1 -> output record ((W1,A), 1) is sent downstream
input record (A, 2, 2) of W1 -> output record ((W1,A), 3) is sent downstream
input record (A, 3, 3) of W1 -> output record ((W1,A), 6) is sent downstream
input record (A, 4, 4) of W2 -> output record ((W2,A), 4) is sent downstream
With reduce().suppress(), the result are buffered until the window closes. The result would be:
input record (A, 1, 1) of W1 -> no output
input record (A, 2, 2) of W1 -> no output
input record (A, 3, 3) of W1 -> no output
input record (A, 4, 4) of W2 -> output record ((W1,A), 6) is sent downstream
Note that for the case without suppress() I assumed that the cache is switched off with cache.max.bytes.buffering = 0 . With cache.max.bytes.buffering > 0 (default is 10MB), the cache will buffer output records of a KTable and once the cache is full, it will output the record with the key that was least recently updated.
Related
Problem Statment:
Topic 1: "key = empId, value = empname, deptName, ..."
Topic 2: "key = deptName, value = deptName"
I need the data from Topic 1 where the deptName(value attribute in topic 1) is equal to key of Topic 2.
Steps:
Create a stream from Topic 1, group it by deptName, and do aggregatation.
It will return Ktable (key =deptName, value = "empId1,empId2,empId3 ..")
Create a stream from Topic 2 (key ="deptName" value = "deptName")
Perform a left join operation on Ktable (Step 1) and KSteam (Step2). (KStream-Ktable)
And join returns desired result.
Everything works as expected in single partition, however, after switching to multiple partitions, join doesn't return any data.
Step 1:
KGroupedStream<String, Object> groupedStream = adStream.groupBy((key, value) -> value.getOrganizationId().toString());
groupedStream
.aggregate(() -> (new String()),
(aggKey, newValue, aggValue) -> addCurrentValue(aggValue,
String.valueOf(newValue.getOriginId())),
Materialized.as("aggregated-stream-store").with(strSerde, strSerde))
.toStream().to(Constant.AD_AGGREGATED_DATA, Produced.with(strSerde, strSerde));
Step 2:
KStream<String, String> swgOrgStream = builder.stream(Constant.SWG_ORG_TOPIC,Consumed.with(strSerde, strSerde));
Step 3:
KStream<String, String> filteredOrgStream = swgOrgStream.leftJoin(aggregatedTable,
(leftValue, rightValue) -> rightValue);
I use two different retention time for two different KTable, and it works with RocksDB States and changelog Kafka Topics.
KTable is generated from KStream and groupBy and then windowedBy.
I believe when joining KStream with windowing, TimeWindows is the same. I'm wondering will there be benefit or drawback if TimeWindows parameters are different, when joining two different KTable windowed by TimeWindows?
code snippet:
final KStream<Integer, String> eventStream = builder.stream("events",
Consumed.with(Serdes.Integer(), Serdes.String())
.withOffsetResetPolicy(Topology.AutoOffsetReset.EARLIEST));
final KTable<Windowed<Integer>, String> eventWindowTable = eventStream.groupByKey()
.windowedBy(TimeWindows.of(Duration.ofSeconds(60)).until(Duration.ofSeconds(100).toMillis()))
.reduce((oldValue, newValue) -> newValue);
final KStream<Integer, String> clickStream = builder.stream("clicks",
Consumed.with(Serdes.Integer(), Serdes.String())
.withOffsetResetPolicy(Topology.AutoOffsetReset.EARLIEST));
final KTable<Windowed<Integer>, String> clickWindowTable = clickStream.groupByKey()
.windowedBy(TimeWindows.of(Duration.ofSeconds(30)).until(Duration.ofSeconds(70).toMillis()))
.reduce((oldValue, newValue) -> newValue);
final KTable<Windowed<Integer>, String> join = eventWindowTable.leftJoin(clickWindowTable,
(event, click) -> event + " ; " + click + " ; " + Instant.now()
);
Initially I thought joining two different KTable with different TimeWindows parameters will not work because the joining relies on TimeWindowedKey, a key for the time slot. But after testing, it works as well.
The join is executed because the type of both keys is the same: Windowed<Integer>. The join will of course only produce a result if the keys are the same. Assume you have the following windows (note that only the window start timestamp is stored for TimeWindows):
eventWindowTable: <A,0> <A,60>
clickWindowTable: <A,0> <A,30> <A,60> <A,90>
For this case, only <A,0> and <A,60> would join. Hence, having different windows, does impact your result, because the window start timestamp is part of the key and some windows will never join (eg, <A,30> and <A,90> in our example).
As the title shows,Java Stream code like follow:
StreamsBuilder builder = new StreamsBuilder();
KStream<String, City> citesStream = builder.stream("cities"
, Consumed.with(Serdes.String(), SerdesFactory.serdesFrom(City.class)));
citesStream.filter((name, city) -> city.getParentId() != 0).to("citiesExcludeProvince"
, Produced.with(Serdes.String(), SerdesFactory.serdesFrom(City.class)));
KTable<String, City> allCityTable = builder.table("citiesExcludeProvince"
, Consumed.with(Serdes.String(), SerdesFactory.serdesFrom(City.class)));
I want filter some city and save to anther Kafka topic and then read it as a KTable for join as follow:
KStream<String, City> provinceStream = citesStream
.filter((name, city) -> city.getParentId() == 0);
provinceStream.leftJoin(allCityTable, (province, city) -> {
System.out.println(JsonUtil.objectToJson(province));
System.out.println(JsonUtil.objectToJson(city));
if (province != null && city != null) {
if (city.getParentId() == province.getId()) {
if (province.getChildren() == null) {
province.setChildren(Lists.newArrayList());
}
province.getChildren().add(city);
}
}
return province;
}).to("provinceWithCity", Produced.with(Serdes.String(), SerdesFactory.serdesFrom(City.class)));
But i get the citiesExcludeProvince topic is alway empty.Where is the error?
Can KStream.to() and StreamsBuilder.table() using the same topic and in the same StreamsBuilder in Kafka Stream?
Yes, you can use an input topic for StreamsBuilder.table() that is an output topic from KStream.to(). StreamsBuilder doesn't allow certain types of cycles, but these—that run through some topic—are allowed. In this regard, I don't think there's anything wrong with your code.
I want filter some city and save to anther Kafka topic and then read it as a KTable for join as follow ... But i get the citiesExcludeProvince topic is alway empty.Where is the error?
There are several problems with your code:
Cities that are arriving into the join are not keyed by province ID. So the join will never happen.
If the cities were keyed by province ID, every city that arrived to the table on the right would override any previous city that arrived on the right. This is because a table is a changelog of values by key. If there are multiple cities that belong to a province in a stream keyed by province ID, in the table you will only see the last one to arrive.
The right-side table doesn't trigger computation. This is a KStream-KTable join, and the semantics of such a join are the only events on the left cause processing. Events on the right are merely stored in the table. (On a related note, you can't really use KStream-KTable joins to process historical data. When you turn on your Kafka Streams application, it has a consumer that reads all your input topics. If it reads the topic that creates provinceStream before the contents of allCityTable, then your provinces won't find anything in the allCityTable because it will still be empty.)
The left-side will never be null (you don't have to do that check).
I think this is what you are looking for:
// Step 1
KTable<String, ArrayList<City>> citiesByProvince = citesStream
.filter((name, city) -> city.getParentId() != 0)
.groupBy((k, v) -> v.getParentId())
.aggregate(ArrayList::new,
(k, v, a) -> {
a.add(v);
return a;
});
// Step 2
provinceStream
.groupBy((k, v) -> v.getId())
.reduce((a, b) -> a)
.join(citiesByProvince, (province, cities) -> {
province.setChildren(cities);
return province;
});
Step 1: aggregate all cities by province ID into a list. The resulting list is keyed by province ID.
Step 2: turn the provinces into a table keyed by province ID (you could do this equivalently by writing the contents of provinceStream to a topic and then using StreamBuilder.table(), but groupBy()->reduce() does the same thing here) and then performs the join.
Unlike your KStream-KTable join, the KTable-KTable join is not sensitive to the order in which records arrive from the underyling consumer, so you'll get deterministic results.
I'm having some issue with how the following topology behaves:
String topic = config.topic();
KTable<UUID, MyData> myTable = topology.builder().table(UUIDSerdes.get(), GsonSerdes.get(MyData.class), topic);
// Receive a stream of various events
topology.eventsStream()
// Only process events that are implementing MyEvent
.filter((k, v) -> v instanceof MyEvent)
// Cast to ease the code
.mapValues(v -> (MyEvent) v)
// rekey by data id
.selectKey((k, v) -> v.data.id)
.peek((k, v) -> L.info("Event:"+v.action))
// join the event with the according entry in the KTable and apply the state mutation
.leftJoin(myTable, eventHandler::handleEvent, UUIDSerdes.get(), EventSerdes.get())
.peek((k, v) -> L.info("Updated:" + v.id + "-" + v.id2))
// write the updated state to the KTable.
.to(UUIDSerdes.get(), GsonSerdes.get(MyData.class), topic);
My Issue happens when i receive different events at the same time. As my state mutation is done by the leftJoin and then written by the to method. I can have the following occuring if event 1 and 2 are received at the same time with the same key:
event1 joins with state A => state A mutated to state X
event2 joins with state A => state A mutated to state Y
state X written to the KTable topic
state Y written to the KTable topic
Because of that, state Y doesn't have the changes from event1, so I lost data.
Here's in terms of logs what I see (the Processing:... part is logged from inside the value joiner):
Event:Event1
Event:Event2
Processing:Event1, State:none
Updated:1-null
Processing:Event2, State:none
java.lang.IllegalStateException: Event2 event received but we don't have data for id 1
Event1 can be considered as the creation event: it will create the entry in the KTable so it doesn't matter if the state is empty. Event2 though needs to apply it's changes to an existing state but it doesn't find any because the first state mutation still hasn't been written to the KTable (it's still hasn't been processed by the to method)
Is there anyway to make sure that my leftJoin and my writes into the ktable are done atomically ?
Thanks
Update & current solution
Thanks to the response of #Matthias I was able to find a solution using a Transformer.
Here's what the code looks like:
That's the transformer
public class KStreamStateLeftJoin<K, V1, V2> implements Transformer<K, V1, KeyValue<K, V2>> {
private final String stateName;
private final ValueJoiner<V1, V2, V2> joiner;
private final boolean updateState;
private KeyValueStore<K, V2> state;
public KStreamStateLeftJoin(String stateName, ValueJoiner<V1, V2, V2> joiner, boolean updateState) {
this.stateName = stateName;
this.joiner = joiner;
this.updateState = updateState;
}
#Override
#SuppressWarnings("unchecked")
public void init(ProcessorContext context) {
this.state = (KeyValueStore<K, V2>) context.getStateStore(stateName);
}
#Override
public KeyValue<K, V2> transform(K key, V1 value) {
V2 stateValue = this.state.get(key); // Get current state
V2 updatedValue = joiner.apply(value, stateValue); // Apply join
if (updateState) {
this.state.put(key, updatedValue); // write new state
}
return new KeyValue<>(key, updatedValue);
}
#Override
public KeyValue<K, V2> punctuate(long timestamp) {
return null;
}
#Override
public void close() {}
}
And here's the adapted topology:
String topic = config.topic();
String store = topic + "-store";
KTable<UUID, MyData> myTable = topology.builder().table(UUIDSerdes.get(), GsonSerdes.get(MyData.class), topic, store);
// Receive a stream of various events
topology.eventsStream()
// Only process events that are implementing MyEvent
.filter((k, v) -> v instanceof MyEvent)
// Cast to ease the code
.mapValues(v -> (MyEvent) v)
// rekey by data id
.selectKey((k, v) -> v.data.id)
// join the event with the according entry in the KTable and apply the state mutation
.transform(() -> new KStreamStateLeftJoin<UUID, MyEvent, MyData>(store, eventHandler::handleEvent, true), store)
// write the updated state to the KTable.
.to(UUIDSerdes.get(), GsonSerdes.get(MyData.class), topic);
As we're using the KTable's KV StateStore and applying changes directly in it through the put method events shoudl always pick up the updated state.
One thing i'm still wondering: what if I have a continuous high throughput of events.
Could there still be a race condition between the puts we do on the KTable's KV store and the writes that are done in the KTable's topic ?
A KTable is sharded into multiple physical stores and each store is only updated by a single thread. Thus, the scenario you describe cannot happen. If you have 2 records with the same timestamp that both update the same shard, they will be processed one after each other (in offset order). Thus, the second update will see the state of after the first update.
So maybe you just did describe your scenario not correctly?
Update
You cannot mutate the state when doing a join. Thus, the expectation that
event1 joins with state A => state A mutated to state X
is wrong. Independent of any processing order, when event1 joins with state A, it will access state A in read only mode and state A will not be modified.
Thus, when event2 joins, it will see the same state as event1. For stream-table join, the table state is only updated when new data is read from the table-input-topic.
If you want to have a shared state that is updated from both inputs, you would need to build a custom solution using transform():
builder.addStore(..., "store-name");
builder.stream("table-topic").transform(..., "store-name"); // will not emit anything downstream
KStream result = builder.stream("stream-topic").transform(..., "store-name");
This will create one store that is shared by both processors and both can read/write as they wish. Thus, for the table-input you can just update the state without sending anything downstream, while for the stream-input you can do the join, update the state, and send a result downstream.
Update 2
With regard to the solution, there will be no race condition between the updates the Transformer applies to the state and records the Transformer processes after the state update. This part will be executed in a single thread, and records will be processed in offset-order from the input topic. Thus, it's ensured that a state update will be available to later records.
{
var history: RDD[(String, List[String]) = sc.emptyRDD()
val dstream1 = ...
val dstream2 = ...
val historyDStream = dstream1.transform(rdd => rdd.union(history))
val joined = historyDStream.join(dstream2)
... do stuff with joined as above, obtain dstreamFiltered ...
dstreamFiltered.foreachRDD{rdd =>
val formatted = rdd.map{case (k,(v1,v2)) => (k,v1) }
history.unpersist(false) // unpersist the 'old' history RDD
history = formatted // assign the new history
history.persist(StorageLevel.MEMORY_AND_DISK) // cache the computation
history.count() //action to materialize this transformation
}
This code logic is working fine for preserving all the previous RDDs which didn't successfully joined and saved for the future batches so that whenever we get a record with corresponding joining key for this RDD , we perform the join but I didn't got how this history is build up.
We can understand how the history builds up in this case by observing how the RDD lineage evolves over time.
We need two pieces of previous knowledge:
RDDs are immutable structures
Operations on RDD can be expressed in functional terms by the function to be applied and references to the input RDDs.
Let's see a quick example, using the classical wordCount:
val txt = sparkContext.textFile(someFile)
val words = txt.flatMap(_.split(" "))
In simplified terms, txt is a HadoopRDD(someFile). words is a MapPartitionsRDD(txt, flatMapFunction). We speak of the lineage of words as the DAG (Direct Acyclic Graph) that is formed of this chaining of operations.: HadoopRDD <-- MapPartitionsRDD.
We can apply the same principles to our streaming operation:
At iteration 0, we have
var history: RDD[(String, List[String]) = sc.emptyRDD()
// -> history: EmptyRDD
...
val historyDStream = dstream1.transform(rdd => rdd.union(history))
// -> underlying RDD: rdd.union(EmptyRDD)
join, filter
// underlying RDD: ((rdd.union(EmptyRDD).join(otherRDD)).filter(pred)
map
// -> underlying RDD: ((rdd.union(EmptyRDD).join(otherRDD)).filter(pred).map(f)
history.unpersist(false)
// EmptyRDD.unpersist (does nothing, it was never persisted)
history = formatted
// history = ((rdd.union(EmptyRDD).join(otherRDD)).filter(pred).map(f)
history.persist(...)
// history marked for persistence (at the next action)
history.count()
// ((rdd.union(EmptyRDD).join(otherRDD)).filter(pred).map(f).count()
// cache result of: ((rdd.union(EmptyRDD).join(otherRDD)).filter(pred).map(f)
At iteration 1, we have (adding rdd0, rdd1 as iteration index):
val historyDStream = dstream1.transform(rdd => rdd.union(history))
// -> underlying RDD: rdd1.union(((rdd0.union(EmptyRDD).join(otherRDD0)).filter(pred).map(f))
join, filter
// underlying RDD: ((rdd1.union(((rdd0.union(EmptyRDD).join(otherRDD0)).filter(pred).map(f)).join(otherRDD1)).filter(pred)
map
// -> underlying RDD: ((rdd1.union(((rdd0.union(EmptyRDD).join(otherRDD0)).filter(pred).map(f)).join(otherRDD1)).filter(pred).map(f)
history.unpersist(false)
// history0.unpersist (marks the previous result for removal, we used it already for our computation above)
history = formatted
// history1 = ((rdd1.union(((rdd0.union(EmptyRDD).join(otherRDD0)).filter(pred).map(f)).join(otherRDD1)).filter(pred).map(f)
history.persist(...)
// new history marked for persistence (at the next action)
history.count()
// ((rdd1.union(((rdd0.union(EmptyRDD).join(otherRDD0)).filter(pred).map(f)).join(otherRDD1)).filter(pred).map(f).count()
// cache result sothat we don't need to compute it next time
This iterative process goes on with each iteration.
As we can see, the graph representing the RDD computation keeps on growing. cache reduces the cost of making all calculations each time. checkpoint is needed every so often to write a concrete computed value of this growing graph so that we can use it as baseline instead of having to evaluate the whole chain.
An interesting way to see this process in action is by adding a line within the foreachRDD to inspect the current lineage:
...
history.unpersist(false) // unpersist the 'old' history RDD
history = formatted // assign the new history
println(history.toDebugString())
...