How to calculate DAU/MAU using KSQL? - etl

Question

I'm a messenger developer and trying to calculate DAU/MAU using an event stream of user's requests using KSQL.
I've tried to calculate it using the following query:
CREATE TABLE ACTIVE_USER_ACTIONS_BY_1_HOUR WITH (
KAFKA_TOPIC='active-user-actions-by-1-hour'
) AS
SELECT
MCCU.UID AS UID,
COUNT(MCCU.UID) AS ACTIVITY_COUNT
FROM METRICS_REQUESTS MR
JOIN METRICS_CONTEXT_CID_UID MCCU ON MCCU.CID = MR.CID
WINDOW TUMBLING (SIZE 1 HOUR)
WHERE
MR.REQ_NAME = 'SendMessage' OR
MR.REQ_NAME = 'UpdateMessage'
GROUP BY MCCU.UID;
I'm getting the following results:
{
"order": 3,
"ROWTIME": 1570095657670,
"ROWKEY": "1365010623 : Window{start=1570093200000 end=-}",
"UID": 1365010623,
"ACTIVITY_COUNT": 3
}
{
"order": 1,
"ROWTIME": 1570095651905,
"ROWKEY": "1637035978 : Window{start=1570093200000 end=-}",
"UID": 1637035978,
"ACTIVITY_COUNT": 9
}
Don't understand how to map those rows to something like:
{
"ACTIVE_UID_COUNT": 2,
"START": 1570093200000,
"END": null
}

Answer 1

If you want to calculate DAU/MAU you must use TOPKDISTINCT function which counts only distinct occurrences of user_id within incoming events.
You should use sql code like below:
CREATE TABLE dau_1min WITH (KAFKA_TOPIC='dau-1m', VALUE_FORMAT='AVRO') AS
SELECT
country,
WINDOWSTART() AS window_timestamp,
ARRAYLENGTH(TOPKDISTINCT(user_id, 10000000)) AS dau
FROM eventssource
WINDOW TUMBLING (SIZE 1 MINUTE)
GROUP BY country;
As TOPKDISTINCT returns ARRAY of distinct identifiers you must implement custom UDF function i.e. ARRAYLENGTH that just returns size of given array.
After that you will get an object like {country, window_timestamp, dau} which is simple enough to store in some persistent database like pqsql.