How to invert many-to-many relationship? - apache-kafka-streams

Question

I have a compacted Kafka topic that is a stream of entities have the latest representation of that entity in a many-to-many relationship that I'd like to invert.
An example would be a topic of Author objects where the topic key is the Author.id (AAA) and the value is an array of `Book' identifier values:
"AAA" -> {"books": [456]}
When an Author writes a new Book with an ID of 333, a new event with the same key is written to the stream with the updated list of books:
"AAA" -> {"books": [456, 333]}
It is also possible that a Book had multiple Authors so that same Book identifier could appear in another event:
"BBB" -> {"books": [333, 555]}
I'd like to invert this using kafka streams into a stream of Books -> [Author], so the above events would result in something like:
456 -> {"authors": ["AAA"]}
333 -> {"authors": ["AAA", "BBB"]}
555 -> {"authors": ["BBB"]}
When I start my app up again, I want the state to be restored such that if I read in another Author record it will invert the relationship appropriatley. So this:
"CCC" -> {"books": [555]}
would know that "BBB" was also an Author and would emit the updated event:
555 -> {"authors": ["BBB", "CCC"]}
I've been eyeing the GlobalKTable which reads in the full topic state locally, but can't figure out how to get it to invert the relationship and aggregate the values together.
If I could, I think I could join that GlobalKTable with a stream of the events and get the full list of Authors for each Book.

Answer 1

You don't have to use GlobakKTable to achieve your requirement.
In Kafka Streams, internal data redistribution caused by changing key occurs automatically. For example :
orgKStream
.flatMapValues(books -> getBookList) (1)
.map((k,v) -> new KeyValue<>(v, k)) (2)
.groupByKey() (3)
.aggregate(//aggregate author list ) (4)
.toStream(// sink topic) (5)
(1) will change your original topic like below.
<before>
"AAA" -> {"books": [456, 333]}
"BBB" -> {"books": [333, 555]}
<after>
"AAA" -> 456
"AAA" -> 333
"BBB" -> 333
"BBB" -> 555
(2) will replace the key with the value.
<after>
456 -> "AAA"
333 -> "AAA"
333 -> "BBB"
555 -> "BBB"
(3) and (4) will aggregate and generate KTable (and state store)
<after>
456 -> {"authors": ["AAA"]}
333 -> {"authors": ["AAA", "BBB"]}
555 -> {"authors": ["BBB"]}
(5) will write whole records in the table into the given topic.
Now, you have a new topic that contains book as key and author list as values. If you want to have the whole result in one place, now just create GlobalKTable like below.
StreamsBuilder.globalTable(<sink topic>)
If (2) is called (map) and then (3) is called (groupByKey), internal data redistribution via repartition topic will occur. It means that all record that has same book id as a key will be published into the same partition of internal repartition topic. As a result, you will not lose any data for your aggregation.