Materializing the final aggregation value (not state) with ClickHouse - clickhouse

Question

I want to create a materialized view in ClickHouse that stores the final product of an aggregation function. The best practice is to store the state and in query time to calculate the final product but it's too costly to do it in query time for my use case.
Base table:
CREATE TABLE IF NOT EXISTS active_events
(
`event_name` LowCardinality(String),
`user_id` String,
`post_id` String
)
My current materialization:
CREATE MATERIALIZED VIEW IF NOT EXISTS inventory
(
`post_id` String,
`event_name` LowCardinality(String),
`unique_users_state` AggregateFunction(uniq, String)
)
ENGINE = AggregatingMergeTree
ORDER BY (event_name, post_id)
POPULATE AS
SELECT
post_id,
event_name,
uniqState(user_id) unique_users_state
FROM active_events
GROUP BY post_id, event_name
FROM test_sessions
GROUP BY session_id;
And then at query time, I can use uniqMerge to calculate the exact number of users who've done a certain event.
I don't mind a small delay in the materialization but I want the full product to be calculated during ingestion rather than the query.
Here's the query:
SELECT post_id, sumIf(total, event_name = 'click') / sumIf(total, event_name = 'impression') as ctr
FROM (
SELECT post_id, event_name, uniqMerge(unique_users_state) as total
FROM inventory
WHERE event_name IN ('click', 'impression')
GROUP BY post_id, event_name
) as res
GROUP BY post_id
HAVING ctr > 0.1
ORDER BY ctr DESC

Answer 1

It's literally impossible.
Imagine you insert into a table some user_id - 3456, how many uniq? 1, Ok but you cannot store 1, because if you insert 3456, it still should be 1. So CH stores states and they are HLL (hyperloglog) structures and they are not fully aggregated/calculated. Because you may query group by event_name or group by event_name, post_id or without groupby.
Another problem why your query is slow. You did not provide your query so I can only make a guess that the issue is index_granularity and CH reads a lot excessive data from the disk IF you query where event_name = ...
It can be solved like this
CREATE MATERIALIZED VIEW IF NOT EXISTS inventory
(
`post_id` String,
`event_name` LowCardinality(String),
`unique_users_state` AggregateFunction(uniq, String) CODEC(NONE) -- uniq is not compressible in 99% cases
)
ENGINE = AggregatingMergeTree
ORDER BY (event_name, post_id)
Settings index_granularity=256 -- 256 instead of default 8192.
Another approach to use another HLL function because uniq is too heavy.
Try this:
CREATE MATERIALIZED VIEW IF NOT EXISTS inventory
(
`post_id` String,
`event_name` LowCardinality(String),
`unique_users_state` AggregateFunction(uniqCombined64(14), String) CODEC(NONE)
)
ENGINE = AggregatingMergeTree
ORDER BY (event_name, post_id)
Settings index_granularity=256
POPULATE AS
SELECT
post_id,
event_name,
uniqCombined64State(14)(user_id) unique_users_state
FROM active_events
GROUP BY post_id, event_name
select uniqCombined64Merge(14)(unique_users_state)
from inventory
where event_name = ...
Attention: need to use (14) in all 3 places uniqCombined64(14) / uniqCombined64State(14) / uniqCombined64Merge(14)
uniqCombined64(14) is more inaccurate than uniq but able to work 10-100 faster in some cases and with an error rate < 5%.