I am trying following YouTube link with Cobalt 22.lts.4.
https://www.youtube.com/tv/watch?v=oxP9PAyBYSc
It keeps creating keeps creating new session after every 2 minutes and never destroys them.
There is max limit of 16 for number of concurrent sessions. When the session creation fails, I am calling SbDrmSessionUpdateRequestFunc with error status. But then the video playback stops.
Can you please suggest how key rotation use-case need to be implemented in Cobalt?
Related
I need to implement a logic similar to session windows using processor API in order to have a full control over state store. Since processor API doesn't provide windowing abstraction, this needs to be done manually. However, I fail to find the source code for KStreams session window logic, to get some initial ideas (specifically regarding session timeouts).
I was expecting to use punctuate method, but it's a per processor timer rather than per key timer. Additionally SessionStore<K, AGG> doesn't provide an API to traverse the database for all keys.
[UPDATE]
As an example, assume processor instance is processing K1 and stream time is incremented which causes the session for K2 to timeout. K2 may or may not exist at all. How do you know that there exists a specific key (like K2 when stream time is incremented (while processing a different key)? In other words when stream time is incremented, how do you figure out which windows are expired (because you don't know those keys exists)?
This is the DSL code: https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSessionWindowAggregate.java -- hope it helps.
It's unclear what your question is though -- it's mostly statements. So let me try to give some general answer.
In the DSL, sessions are close based on "stream time" progress. Only relying on the input data makes the operation deterministic. Using wall-clock time would introduce non-determinism. Hence, using a Punctuation is not necessary in the DSL implementation.
Additionally SessionStore<K, AGG> doesn't provide an API to traverse the database for all keys.
Sessions in the DSL are based on keys and thus it's sufficient to scan the store on a per-key basis over a time range (as done via findSessions(...)).
Update:
In the DSL, each time a session window is updated, as corresponding update event is sent downstream immediately. Hence, the DSL implementation does not wait for "stream time" to advance any further but publishes the current (potentially intermediate) result right away.
To obey the grace period, the record timestamp is compared to "stream time" and if the corresponding session window is already closed, the record is skipped (cf. https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSessionWindowAggregate.java#L146). I.e., closing a window is just a logical step (not an actually operation); the session will still be stored and if a window is closed no additional event needs to be sent downstream because the final result was sent downstream in the last update to the window already.
Retention time itself must not be handled by the Processor implementation because it's a built-in feature of the SessionStore: internally, the session store maintains so-called "segments" that store sessions for a certain time period. Each time a put() is done, the store checks if old segments can be dropped (based on the timestamp provided by put()). I.e., old sessions are deleted lazily and as bulk deletes (i.e., all session of the whole segment will be deleted at once) as it's more efficient than individual deletes.
I have a requirement as stated # https://kafka.apache.org/21/documentation/streams/developer-guide/dsl-api.html#window-final-results for waiting until window is closed in order to handle late out of order event by buffering it for duration of window.
Per my understanding of this feature is once windowing is created, the window works like wall clock processing, e.g. Creating for 1 hour window, The window starts ticking once first event comes. This 1hr window is closed exactly one hour later and all the events buffered so far will be forwarded to down stream. However, i need to be able to hold this window even longer say conditionally for as long as required e.g. based on state / information in external system such as database.
To be precise my requirement for event forwarding is (windows of 1 hour if external state record says it is good) or (hold for as long as required until external record says it's good and resume tracking of the event until the event make it fully 1hr, disregarding the time when external system is not good)
To elaborate this 2nd condition, e.g. if my window duration 1 1hr , my event starts at 00:00, if on 00:30 it is down and back normal on 00:45, the window should extend until 01:15.
Is it possible to pause and resume the forwarding of events conditionally based on my requirement above ?
Do I have to use transformation / processor and use value store manually to track the first processing time of my event and conditionally forwarding buffered events in punctuator ?
I appreciate all kind of work around and suggestion for this requirement.
the window works like wall clock processing
No. Kafka Streams work on event-time, hence, the timestamps as returned from the TimestampExtractor (by default the embedded record timestamp) are use to advance time.
To be precise my requirement for event forwarding is (windows of 1 hour if external state record says it is good)
This would need a custom solution IMHO.
or (hold for as long as required until external record says it's good and resume tracking of the event until the event make it fully 1hr, disregarding the time when external system is not good)
Not 100% if I understand this part.
Is it possible to pause and resume the forwarding of events conditionally based on my requirement above ?
No.
Do I have to use transformation / processor and use value store manually to track the first processing time of my event and conditionally forwarding buffered events in punctuator ?
I think this might be required.
Check out this blog post, that explains how suppress() work in details, and when it emits based on observed event-time: https://www.confluent.io/blog/kafka-streams-take-on-watermarks-and-triggers
If a user is publishing to a tokbox session and for any reason that same user logs in on a different device or re-opens the session in another browser window I want to stop the 2nd one from publishing.
Luckily, on the metadata for the streams, I am saving the user id, so when there is a list of streams it's easy to see if an existing stream belongs to the user that is logged in.
When a publisher gets initialized the following happens:
Listen for session.on("streamCreated") when this happens, subscribe to new streams
Start publishing
The problem is, when the session gets initialized, there is no way to inspect the current streams of the session to see if this user is already publishing. We don't know what the streams are until the on("streamCreated") callback fires.
I have a hunch that there is an easy solution that I am missing. Any ideas?
I assume that when you said you save the user ID on the stream metadata, that means when you initialize the Publisher, you set the "name" property. That's a great technique.
My idea is slightly a hack, but its the best I can come up with right now. I would solve this problem by essentially breaking up the subscription of streams into 2 phases:
all streams created before this client connection
all streams created after
During #1 I would check each stream's "name" property to see if it belongs to the user at this client connection. If it does, then you know they are entering the session twice and you can set a flag (lets call it "userRejoining". In order to know that #1 is complete, I would set a timer (this is why I call it a hack) for a reasonable amount of time such as 1 second each time a "streamCreated" event arrives, and remove the any previous timer.
Then, if the "userRejoining" flag is not set, the Publisher is initialized and published to the session.
During #2, you just subscribe to any stream that is created.
The downside is that you've now delayed your user experience of publishing by ~1 second everywhere. In larger group scenarios this could be a deal breaker, but in smaller (1:1) types of sessions this should be acceptable. I hope this explanation is clear, and if not I can try to write some sample code for you.
I'm developing an app that has multiple music playlists, a view for each playlist and a player.
I'm making use of the HTML5 History API in my system already (for other functionality) and will be using it further to prevent page reloads between requests and therefore the music stopping on each page.
I am stuck as to the best approach for managing the tracks the player will be playing between views though. At present, as you would expect, a user clicks a link, gets a list of tracks. The tracks are loaded into the player and played sequentially, simple. However, with the continuously playing music i need to ensure the correct playlist of tracks plays in order despite the changing content on the page and it's now dynamic URL.
However, when the user does navigate to another page, presses play on a track in the new playlist, i need to be able to load that item into the player and effectively load in the rest of the playlist while the user continues to navigate.
I am using Redis to store a list of the track id's for a playlist, for speedy referencing to reduce lag between tracks. As a result i have a different Redis set for each playlist. I have also built my Next and Previous Track API calls based on the current track playing, so that the next track from the Redis set can be loaded into the player.
As mentioned though i just can't decide on the best way to reference which playlist is currently playing so the player knows from which Redis set to call the tracks from. My thinking has left me with a few different ideas:
a) HTML5 custom data attributes - i could set the currently playing playlist as a data-attribute on the player and update it as and when. I could then reference this attribute when deciding which Redis set to load my next track from.
Alternatively i could dump the current playlist, and all the tracks (and their attributes) as JSON objects into data attributes on the page. Playlists could be thousands of tracks long though so i'm dismissing this one because the source code would look horrible, let alone the likely performance issues associated. Am i right?
b) LocalStorage - Cross browser support is limited. The more support for this solution the better in this particular case. Despite this i could save the current playlist in the users browser. This has led me to wonder whether it would also be practical to save the tracks JSON objects in LocalStorage too, to prevent the additional DB calls.
c) In Session - I could update the session to store a variable for the currently playing playlist.
d) Redis - I could expand my usage of Redis to save a string that references the name of the current playlist. I could then check it between each Next/Previous track call.
Through writing this question i already have a better idea of which route i am going to take but if anyone has any advice for this scenario then i'd love to hear please.
Thanks.
UPDATE:
I have implemented 95% of a solution that makes use of Redis for this. I am having a few performance issues though with pages taking ~10s to load. Not great at all.
Essentially each user has 2 playlists: Current and Armed. Each request loads the track id's into the Redis Armed playlist and if the play button is pressed on a track, the Current Redis playlist is expired and replaced with the Armed one.
My Next and Previous buttons then just fetch the ID of the next or previous track in the Current playlist and load in the music source for the player. The concept works fine and i'm pleased with it.
However, as mentioned performance is slow between page requests and needs significant improvement. My SQL is optimimised, i'm only pulling out the required attributes and i have SQL indexes where necessary so i'm looking for other alternatives at the moment.
Options i'm considering:
Only populate the Armed playlist if a track is clicked on the new page. This would save additional processing if the user doesn't actually want to listen to one of the new tracks.
Making more use of Redis and storing the lean track objects within the Redis playlists instead of just the track ID - the performance lag is largely between page requests though and not in the actual playing of tracks and navigating a playlist.
Make use of a master Redis playlist that contains all of the applications tracks from which the Current and Armed playlists can pick from. This could be maintained via an hourly rake task and would prevent lengthy DB calls on page requests. I'm just nervous has to how far this would scale in terms of memory usage on the server and the amount of tracks in the DB.
If you need to model client-side application state it's best to have one source of the state. This is the one of the problems client side javascript MV(V)C frameworks attempt to solve. I'm mostly familiar with backbone.js and ember.js so I can speak to those.
Backbone.js
Create a model called Playlist and a collection called Playlists. Add a property to your Playlists collection currentPlaylist that holds the current playlist. Then define a view called PlaylistView and define a render method on it. Wire up event triggers and bindings such that when currentPlaylist changes, the playlist is automatically re-rendered. This would require moving your template rendering to the client, but you'll probably want to do that anyway to reduce server roundtrips and reduce the load on the server caused by rendering.
Ember.js
Create a controller (similar to backbone collection) with a property called currentPlaylist and populate the controller with all of the playlists represented as Ember.objects. Then in a playlist handlebars template included on the page you can bind to playlists.currentPlaylist and the template will re-render automatically when playlists.currentPlaylist changes.
I'm obviously leaving out the vast majority of the details, but those are best left the the framework documentation, examples and tutorials.
disclaimer: I'm new to client-side frameworks which is part of the reason I left out most of the details. I appreciate anyone who can correct me if I'm in error.
A mix of session and local storage would be a good approach.
But instead of fetching the whole playlist just fetch the X (for example 10) next tracks and maybe also the X previous ones. Once the player gets to the last song, it fetches the next 10 songs, the previous ones can be calculated in the client.
The data model could be just a hash where element [0] is the current song, elements [X] are the next songs and [-X] the previous ones.
Storing the playlist information client-side seems reasonable to me, but you can also use the session to reference the current song and playlist, so when a user comes back or really reloads your site you still get the song without doing a database call.
I would suggest c) In Session.
Session data can be accessed with relative ease on both client and server side.
Filling your Redis cache with specific user data doesn't scale up particularly well.
LocalStorage - correct, this will fail for a large % of users at current date.
CustomData Attributes - just plain messy, as noted.
Just my 2 cents.
I have a certain resouce I want to limit access to. Basically, I am using a session level lock. However, it is getting to be a pain writing JavaScript that covers every possible way a window can close.
Once the user leaves that page I would like to unlock the resouce.
My basic idea is to use some sort of server side timeout, to unlock the resouce. Basically, if I fail to unlock the resource, I want a timer to kick in and unlock the resouce.
For example, after 30 seconds with now update from the clientside, unlock the resouce.
My basic question, is what sort of side trick can I use to do this? It is my understanding, that I can't just create a thread in JSF, because it would be unmanaged.
I am sure other people do this kind of thing, what is the correct thing to use?
Thanks,
Grae
As BalusC right fully asked, the big question is at what level of granularity would you like to do this locking? Per logged-in user, for all users, or perhaps you could get away with locking per request?
Or, and this will be a tougher one, is the idea that a single page request grabs the lock and then that specific page is intended to keep the lock between requests? E.g. as a kind of reservation. I'm browsing a hotel page, and when I merely look at a room I have made an implicit reservation in the system for that room so it can't happen that somebody else reserves the room for real while I'm looking at it?
In the latter case, maybe the following scheme would work:
In application scope, define a global concurrent map.
Keys of the map represent the resources you want to protect.
Values of the map are a custom structure which hold a read write lock (e.g. ReentrantReadWriteLock), a token and a timestamp.
In application scope, there also is a single global lock (e.g. ReentrantLock)
Code in a request first grabs the global lock, and quickly checks if the entry in the map is there.
If the entry is there it is taken, otherwise it's created. Creation time should be very short. The global lock is quickly released.
If the entry was new, it's locked via its write lock and a new token and timestamp are created.
If the entry was not new, it's locked via its read lock
if the code has the same token, it can go ahead and access the protected resource, otherwise it checks the timestamp.
If the timestamp has expired, it tries to grab the write lock.
The write lock has a time-out. When the time-out occurs give up and communicate something to the client. Otherwise a new token and timestamp are created.
This just the general idea. In a Java EE application that I have build I have used something similar (though not exactly the same) and it worked quite well.
Alternatively you could use a quartz job anyway that periodically removed the stale entries. Yet another alternative for that is replacing the global concurrent map with e.g. a JBoss Cache or Infinispan instance. These allow you to define an eviction policy for their entries, which saves you from having to code this yourself. If you have never used those caches though, learning how to set them up and configuring them correctly can be more trouble than just building a simple quartz job yourself.