Suppose I have a basic KafkaStreams application with one topic (having multiple partitions) and one processor type handling the messages as follows:
builder.stream(topic)
.process(() -> new MyProcessor());
Can the following scenario occur or not? For a particular instance of MyProcessor, say M (i.e. the particular java object obtained through invocation of the processor supplier), and for a particular partition of the topic, say P,
at some time t1, M receives messages from P
at a later point t2, P is revoked from M so M does not receive messages from P anymore (e.g. because an extra worker was started which handles P)
at a later point, t3, M receives messages again from P
I checked the documentation on how stream tasks relate to Kafka topic partitions but I did not find detailed information on how this relates to the construction and deletion of processor instances and/or (un-)assigment of topic partitions to existing processors when rebalancing occurs.
In Kafka Streams, the "unit of processing" is called a stream task.
Tasks can be stateful and/or stateless. When a rebalancing event happens, a task running on one instance (say, M) of your application may be moved to another instance of your application.
There's a 1-1 mapping between topic partitions and stream tasks, which guarantees that one and only one task will process data from a particular partition. For example, if task 3 is responsible to read from and process in partition P, then when task 3 is moved from instance M to another instance M', then M will stop reading P (because it no longer runs the task 3), and M' (where task 3 runs now) will resume/continue to process P.
at some time t1, M receives messages from P
Let's say the stream task responsible for processing the topic partition P is called task(P). At time t1, M happens to be the app instance that is running task(P). This is the situation at point #1 above.
at a later point t2, P is revoked from M so M does not receive messages from P anymore (e.g. because an extra worker was started which handles P)
Here, another instance of the application (you referred to this instance as "extra worker") becomes responsible for running task(P). Here, task(P) will be migrated automatically from the original app instance M to the new instance M'. Any state that was managed by task(P) (e.g., when the task was doing a stateful operation such as a join or an aggregation) will of course be migrated together with the task. When task(P) is being migrated, also the responsibility for reading from and processing the topic partition P will move from app instance M to M'.
Perhaps don't think too much in terms of "which app instance is handling topic partition P?" Rather, a particular partition is always handled by a particular stream task, and stream tasks may move across app instances. (Of course, Kafka's Streams API will prevent unnecessary task migrations to ensure the processing of your application stays efficient.)
at a later point, t3, M receives messages again from P
This means that, at time t3, M has been assigned the task task(P) again as a result of another rebalancing event -- perhaps because the other app instance M' was taken down, or something else happened that required task migration.
Asked in the comments to this answer: It would also be useful though to have a sentence or two about the state migration. It's not like the binary/physical data is taken from one RocksDB instance and passed over to another. Obviously, the state is re-built based on fault-tolerance mechanism.
Stateful tasks use state stores to, well, persistent state information. These state stores are fault-tolerant. The source of truth for these state stores is Kafka itself: any changes to a state store (e.g., incrementing a counter) are backed up to Kafka in a streaming manner—similar to storing the CDC stream of a database table into Kafka topics (these are normal topics but often called 'changelog topics'). Then, when a task dies or is migrated to another container/VM/machine, the task's state store(s) are restored in the task's new container/VM/machine by reading from Kafka (think: streaming-backup / streaming-restore). This restores the state store(s) to exactly how they looked like on the original container, without any data loss or duplication.
A stream task uses RocksDB to materialize a state store locally (like in the task's container) for optimization purposes. Think of these local RocksDB instances as caches that are ok to be lost with regards to data safety, because the durable storage of the state data is Kafka, as described above.
Related
We are planning to use suppress operator over Session Windowed KTable.
We are wondering about fault-tolerance when using suppress operator.
We understand that buffer is used to store events/aggregations until the window closes.
Now let us say a rebalance has happened, and active task is moved to different machine. We are wondering what happens to this (in-memory ?) buffer.
Let us say we are tracking click count by user. And we configured session window's in-activity period to be 3 minutes, and session window has started for a key alice, and aggregations happened for that key for 2 minutes. For example in buffer we have (alice -> 5) entry representing that alice had made 5 clicks in this session so far.
And say there is no activity after that from alice.
If things are working fine , then once the session is over, downstream processor will get event alice -> 5 .
But what if there is rebalance now, and active task that is maintaining session window for alice is moved to new machine ?
Since there is no further activity from alice, will downstream processor which is running on new machine miss this event alice ->5 ?
The suppress operator provides fault tolerance similarly to any other state store in Streams. Although the active data structure is in memory, the suppression buffer maintains a changelog (an internal Kafka topic).
So, when you have that rebalance, the previous active task flushes its state to the changelog and discards the in-memory buffer. The new active task re-creates the state by replaying the changelog topic, resulting in the exact same buffered contents as if there had been no rebalance.
In other words, just like in-memory state stores, the suppression buffer is made durable (in a Kafka topic) even though it is not persistent (on the local disk).
Does that make sense?
Let's say that we have microservice A (MS A) and Microservice B (MS B).
MS B has data about Products. MS A needs the productnames of MS B.
Each time a product is added, updated or deleted, MS B puts a message on a message queue.
MS A is subscribed to that queue, so it can updated it's own internal state.
Now my question:
How do we fill the internal state of MS A when we deploy it to production the first time?
I couldn't find any documentation about the pros and cons of the possible solutions.
I could think of:
Export/import on database level.
Pros: not much work.
Cons: can miss data if during export/import changes to the data of MS A are made.
Implement calls for GetData and GetDataChangedSince
Pros: failsafe
Cons: a lot of work
Are there any other options? Are there any other pros/cons?
You could use the following workflow:
prepare the microservice B to push the events to the queue or stop it if it is already pushing to the queue; instead, it pushes to a circular buffer (a buffer that is rewritten when full) and waits for a signal from microservice A
deploy the microservice A into production servers but you don't reference it from nowhere; it just runs, waiting for events in the queue
run a script that get all product names from microservice B and push them into the queue as a simulated event; when it finishes the product names it signals the microservice B (optionally telling the date or sequence number or whatever de-duplication technique you have to detect duplicate events)
microservice B then copy the events from the buffer newer that the last pushed by microservice A (or it finds out itself from the queue what is the last one) into the queue and then ignores the buffer and continue to work as normally.
It sounds like there is a service/API call missing from you architecture. Moving a service into production should be no different than recovering from a a failure and should not require any additional steps. Perhaps the messages should be consumed from the queue by another service that can then be queried for the complete list of products.
I'll try to explain this the best I can.
As I store my data that I receive from my ActiveMQ queue in several distinct locations, I have decided to build a composite Queue so I can process the data for each location individually.
The issue I am running into is that I currently have the Queue in a production environment. It seems that changing a queue named A to a composite Queue also called A having virtual destinations named B and C causes me to lose all the data on the existing Queue. It does not on start-up forward the previous messages. Currently, I am creating a new CompositeQueue with a different name, say D, which forwards data to B and C. Then I have some clunky code that prevents all connections until I have both a) updated all the producers to send to D and b) pulled the data from A using a consumer and sent it to D with a producer.
It feels rather messy. Is there any way around this? Ideally I would be able to keep the same Queue name, have all its current data sent to the composite sub-queues, and have the Queue forward only in the end.
From the description given the desired behavior is no possible as message routing on the composite queue works when messages are in-flight and not sometime later when that queue has already stored messages and the broker configuration is changed. You need to consume the past messages from the initial Queue (A I guess it is) and send them onto the destinations desired.
In Spring cloud stream, what exactly is the usage of that property spring.cloud.stream.instanceCount?
I mean if that value become wrong because at a moment one or more micro services instances are down, how could this affect the behavior of our infrastructure?
instanceCount is used to partition data across different consumers. Having one or more services down should not really impact your producers, that's the job of the broker.
So let's say you have a source that sends data to 3 partitions, so you'd have instanceCount=3 and each instance would have it's own partition assigned via instanceIndex.
Each instance would be consuming data, but if instance 2 crashes, 0,1 would still be reading data from the partitions, and source would still be sending data as usual.
Assuming your platform has some sort of recoverability in place, your crashed instance should come back to life and resume it's operations.
What we still don't support is dynamic allocation of partitions on runtime, we are investigating this as a story for a future release.
I have started to use Storm recently but could not find any resources on the net about global grouping option's fault tolerance.
According to my understanding from the documents; while running a topology with a bolt(Bolt A) which is uses global grouping will receive tuples from tasks of Bolt B into the task of Bolt A. As it is using global grouping option, there is only one task of Bolt A in the topology.
The question is as follows: What will happen if we store some historical data of the stream within Bolt A and the worker process dies which contains the task of Bolt A? Meaning that will the data stored in this bolt get lost?
Thanks in advance
Once all the downstream tasks have acked the tuple, it means that they have successfully processed the message and it need not be replayed if there is a shut down. If you are keeping any state in memory, then you should store it in a persistent store. Message should be acked when the state change due to the message has been persisted.