Session Window preventing GroupByKey from working - session

Question

I have an incoming stream of events, each of which already has an associated sessionId from another process.
All I wish to do is combine these events into a single session object using a custom CombineFn.
During development, I'm using a bounded dataset that reads from a file and the following code seems to work:
input.apply(ParDo.named("ParseEvent").of(new ParseEventFn()))
.setCoder(KvCoder.of(StringUtf8Coder.of(), AvroCoder.of(Event.class)))
.apply(GroupByKey.<String, Event>create())
.apply(Combine.groupedValues(new SessionAccumulator()))
The above code (with input/output handling) will output a series of sessions with multiple events in each.
{sessionId: 1, events: [event1,event2,event3]}
{sessionId: 2, events: [event4,event5]}
But in order for this to work on an unbounded dataset, I need to apply a Windowing function, which in this case is a SessionWindow.
input.apply(ParDo.named("ParseEvent").of(new ParseEventFn()))
.setCoder(KvCoder.of(StringUtf8Coder.of(), AvroCoder.of(Event.class)))
.apply(Window.<KV<String, Event>>into(Sessions.withGapDuration(Duration.standardMinutes(30))))
.apply(GroupByKey.<String, Event>create())
.apply(Combine.groupedValues(new SessionAccumulator()))
In that case the only new code is the Windowing function, and rather than rolling up the events, I get each event in it's own session, like this:
{sessionId: 1, events: [event1]}
{sessionId: 1, events: [event2]}
{sessionId: 1, events: [event3]}
{sessionId: 2, events: [event4]}
{sessionId: 2, events: [event5]}
Any idea why this is happening?
EDIT: I should add, the ParseEventFn is applying a timestamp to the PCollection using context.outputWithTimestamp(), and that timestamp seems to be correct.

Answer 1

Digging into it further, in my case the issue was that my core assumption that the timestamps were correct, was wrong.
The timestamps I was applying before the windowing were wrong.
The Windowing was doing exactly what it should, but I had set my timestamps too far apart and it was creating separate sessions for each event.
Oops

Answer 2

In your case, you could possibly write your own WindowFn. If you set the keys to be the session IDs then a large gap duration also works, but it doesn't reflect the nature of your data and computation quite as well.
The ingredients to your WindowFn would be:
your own subclass of BoundedWindow, in this case you would make a window type that contained the session ID in a field
assignWindows, where you would assign each element to a window identified by the session ID. The length of the window still matters, as it controls when the window expires and is garbage collected.
mergeWindows, where you would merge all windows that have the same session ID. They wouldn't have to fall within any particular gap duration.
Another thing you'll need to be careful of is that the watermark that governs the garbage collection of these windows is determined by the source of your unbounded stream of events. So setting the timestamps in your ParDo.of(new ParseEventFn()) will be too late to influence the watermark. You may have data dropped that you'd like to keep.