For example, a windowOperator has 10 parallelisms, and there are 1000 different keys, then each parallelism will process 100 keys, are these 100 keys'states shared or isolated?
For the most part, throughout Flink the state for different keys is isolated. This is done deliberately, so that the state can be resharded when rescaling the cluster.
In a KeyedProcessFunction, or in a ProcessWindowFunction for a keyed window, whenever you access or update state, there is a key implicitly in context. In the case of a KeyedProcessFunction this is key for the event or timer being processed; in the case of a ProcessWindowFunction it's the key associated with the events in the iterable provided. Any state accesses or updates performed will be scoped to that one key. There is no way to access or modify the state associated with other keys.
Related
I need to implement a logic similar to session windows using processor API in order to have a full control over state store. Since processor API doesn't provide windowing abstraction, this needs to be done manually. However, I fail to find the source code for KStreams session window logic, to get some initial ideas (specifically regarding session timeouts).
I was expecting to use punctuate method, but it's a per processor timer rather than per key timer. Additionally SessionStore<K, AGG> doesn't provide an API to traverse the database for all keys.
[UPDATE]
As an example, assume processor instance is processing K1 and stream time is incremented which causes the session for K2 to timeout. K2 may or may not exist at all. How do you know that there exists a specific key (like K2 when stream time is incremented (while processing a different key)? In other words when stream time is incremented, how do you figure out which windows are expired (because you don't know those keys exists)?
This is the DSL code: https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSessionWindowAggregate.java -- hope it helps.
It's unclear what your question is though -- it's mostly statements. So let me try to give some general answer.
In the DSL, sessions are close based on "stream time" progress. Only relying on the input data makes the operation deterministic. Using wall-clock time would introduce non-determinism. Hence, using a Punctuation is not necessary in the DSL implementation.
Additionally SessionStore<K, AGG> doesn't provide an API to traverse the database for all keys.
Sessions in the DSL are based on keys and thus it's sufficient to scan the store on a per-key basis over a time range (as done via findSessions(...)).
Update:
In the DSL, each time a session window is updated, as corresponding update event is sent downstream immediately. Hence, the DSL implementation does not wait for "stream time" to advance any further but publishes the current (potentially intermediate) result right away.
To obey the grace period, the record timestamp is compared to "stream time" and if the corresponding session window is already closed, the record is skipped (cf. https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSessionWindowAggregate.java#L146). I.e., closing a window is just a logical step (not an actually operation); the session will still be stored and if a window is closed no additional event needs to be sent downstream because the final result was sent downstream in the last update to the window already.
Retention time itself must not be handled by the Processor implementation because it's a built-in feature of the SessionStore: internally, the session store maintains so-called "segments" that store sessions for a certain time period. Each time a put() is done, the store checks if old segments can be dropped (based on the timestamp provided by put()). I.e., old sessions are deleted lazily and as bulk deletes (i.e., all session of the whole segment will be deleted at once) as it's more efficient than individual deletes.
My idea is to keep track of states of a domain object by spring statemachine. i.e. statemachine defines how to transit states of the domain object. When the events are persisted/restored to/from the event store, the state of the domain object can be (re)generated by sending events to the statemachine.
However, it seems that creating a statemachine object is relatively expensive, it's not that performant to create a state-machine object whenever a state transition happened on a domain object. If I only maintain a statemachine object, I would worry about concurrency problems. One approach is to have a 'statemachine-pool', but it gets messy if I have to create statamachines for multiple different domain objects.
So is it a good idea to apply spring statemachine with event sourcing pattern?
Provided that all the transitions are based on events I would say that it is a pretty good idea, yes.
The fundamental idea of Event Sourcing is that of ensuring every change to the state of an application is captured in an event object, and that these event objects are themselves stored in the sequence they were applied for the same lifetime as the application state itself.
The main point about event sourcing is that you store the events leading to a particular state - instead of just storing the current state - so that you can replay them up to a given point of time.
Thus, using event sourcing has no impact on how you create your state machines.
However, it seems that creating a state-machine object is relatively expensive, it's not that performant to create a state-machine object whenever a state transition happened on a domain object.
Creating a state-machine every time there is a state transition is not related with event sourcing. Would you do it differently if you were only storing the current state? You'd still need to either create the state-machine from the last stored state - or look it up in a cache or a pool - before you could apply the transition.
The only performance hit derived from using event sourcing would be that of replaying the transitions from the beginning in order to reach the current state. Now, if this is costly you can use snapshots to minimize the amount of transitions that must be replayed.
I have a Guava cache which I would like to expire after X minutes have passed from the last access on a key. However, I also periodically do an action on all the current key-vals (much more frequently than the X minutes), and I wouldn't like this to count as an access to the key-value pair, because then the keys will never expire.
Is there some way to read the value of the keys without this influencing the internal state of the cache? ie cache._secretvalues.get(key) where I could conceivably subclass Cache to StealthCache and do getStealth(key)? I know relying on internal stuff is non-ideal, just wondering if it's possible at all. I think when I do cache.asMap.get() it still counts as an access internally.
From the official Guava tutorials:
Access time is reset by all cache read and write operations (including
Cache.asMap().get(Object) and Cache.asMap().put(K, V)), but not by
containsKey(Object), nor by operations on the collection-views of
Cache.asMap(). So, for example, iterating through cache.entrySet()
does not reset access time for the entries you retrieve.
So, what I would have to do is iterate through the entrySet instead to do my stealth operations.
I am trying to establish a barrier between to different processes in Windows. They are essentially two copies of the same process (Running them as two separate threads instead of processes is not an option).
The idea is to place barriers at different stages of the program, to make sure that both processes start each stage at the same time.
What is the most efficient way of implementing this in Windows?
Use a named event (see CreateEvent and WaitForSingleObject API functions). You would need two events per barrier - each event created in another instance of the application. Then both instances wait for each other's event. Of course, these events can be reused later for another barrier.
There exists one complexity, though - as event names are globally unique (let's say so for simplicity), each event would have a different name, maybe prefixed by the instance's process ID. So each instance of the application would have to get another instance's ID in order to find the name of the event created by another instance.
If you have a windowed application, you can broadcast a message which will inform the second instance of the application about an existence of the first instance.
I am really new to Redis and have been using it along with my Ruby on Rails (Rails 2.3 and Ruby 1.8.7) application using the redis gem for simple tagging functionality as a key value store. I recently realized that I could use it to maintain a user activity feed as well.
The thing is I need the tagging data (stored as key => Sets) in memory and its extremely important to determine results for tagging related operations, where as for the activity feed the data could be deleted on a first in first out basis. Assuming I store X number of activities for every user
Is it possible that I could namespace the redis data sets and have one remain permanently in memory and have the other stay temporarily in the memory. What is the general approach when one uses unrelated data sets that need to have different durations of survival in memory.
Would really appreciate any help on this.
You do not need to define a specific namespace for this. With Redis, you can use the EXPIRE command to set a timeout on a key by key basis.
The general policy regarding key expiration is defined in the configuration file:
# MAXMEMORY POLICY: how Redis will select what to remove when maxmemory
# is reached? You can select among five behavior:
#
# volatile-lru -> remove the key with an expire set using an LRU algorithm
# allkeys-lru -> remove any key accordingly to the LRU algorithm
# volatile-random -> remove a random key with an expire set
# allkeys->random -> remove a random key, any key
# volatile-ttl -> remove the key with the nearest expire time (minor TTL)
# noeviction -> don't expire at all, just return an error on write operations
#
For your purpose, the volatile-lru policy should be set.
You just have to call EXPIRE on the keys you want to be volatile, and let Redis evict them. However please note it is difficult to guarantee that the oldest keys will be evicted first once the timeout has been triggered. More explanations here.
For your specific use case however, I would not use key expiration but rather try to simulate capped collections. If the activity feed for a given user is represented as a list of objects, it is easy to LPUSH the activity objects, and use LTRIM to limit the size of the list. You get FIFO behavior and keep memory consumption under control for free.
UPDATE:
Now, if you really need to isolate data, you have two main possibilities with Redis:
using two distinct databases. Redis database are identified by an integer, and you can have several of them per instance. Use the select command to switch between databases. Databases can be used to isolate data, but not to assign them different properties (like an expiration policy for instance).
using two distinct instances. An empty Redis instance is a very light process. So several of them can be started without any problem. It is actually the best and the more scalable way to isolate data with Redis. Each instance can have its own policies (including eviction policy). The clients should open as many connections as instances.
But again, you do not need to isolate data to implement your eviction policy requirements.