Get top n rows without order by operator in Clickhouse - clickhouse

Question

I had a table
CREATE TABLE StatsFull (
Timestamp Int32,
Uid String,
ErrorCode Int32,
Name String,
Version String,
Date Date MATERIALIZED toDate(Timestamp),
Time DateTime MATERIALIZED toDateTime(Timestamp)
) ENGINE = MergeTree() PARTITION BY toMonday(Date)
ORDER BY Time SETTINGS index_granularity = 8192
And I needed to get top 100 Names with unique Uids or top 100 ErrorCodes.
The obvious query is
SELECT Name, uniq(PcId) as cnt FROM StatsFull
WHERE Time > subtractDays(toDate(now()), 1)
GROUP BY Name ORDER BY cnt DESC LIMIT 100
But data was too big so I created an AggregatingMergeTree because I did not need data filtering by hour (just by date).
CREATE MATERIALIZED VIEW StatsAggregated (
Date Date,
ProductName String,
ErrorCode Int32,
Name String,
Version String,
UniqUsers AggregateFunction(uniq, String),
) ENGINE = AggregatingMergeTree() PARTITION BY toMonday(Date)
ORDER BY
(
Date,
ProductName,
ErrorCode,
Name,
Version
) SETTINGS index_granularity = 8192 AS
SELECT
Date,
ProductName,
ErrorCode,
Name,
Version,
uniqState(Uid) AS UniqUsers,
FROM
StatsFull
GROUP BY
Date,
ProductName,
ErrorCode,
Name,
Version
And my current query is:
SELECT Name FROM StatsAggregated
WHERE Date > subtractDays(toDate(now()), 1)
GROUP BY Name
ORDER BY uniqMerge(UniqUsers) DESC LIMIT 100
The query was working fine, however eventually data rows in a day became more and now it too greedy by memory. So I am looking for some optimization.
I have found the function topK(N)(column) that returns an array of the most frequent values in the specified column but it isn't what I need.

Answer 1

I would suggest to the next points:
where possible prefer use SimpleAggregateFunction instead of AggregateFunction
use uniqCombined/uniqCombined64 that "consumes several times less memory" in compare with uniq
reduce the count of dimensions in aggregated-view (it looks like ProductName and Version can be omitted)
CREATE MATERIALIZED VIEW StatsAggregated (
Date Date,
Name String,
ErrorCode Int32
UniqUsers AggregateFunction(uniq, String),
) ENGINE = AggregatingMergeTree()
PARTITION BY toMonday(Date)
ORDER BY (Date, Name, ErrorCode) AS
SELECT Date, Name, ErrorCode, uniqState(Uid) AS UniqUsers,
FROM StatsFull
GROUP BY Date, Name, ErrorCode;
adding extra 'heuristic' constraints to when-clause of resulting query
SELECT Name, uniqMerge(UniqUsers) uniqUsers
FROM StatsAggregated
WHERE Date > subtractDays(toDate(now()), 1)
AND uniqUsers > 12345 /* <-- 12345 is 'heuristic' number that you evaluate based on your data */
AND ErrorCode = 0 /* apply any other conditions to narrow the result set as short as possible */
GROUP BY Name
ORDER BY uniqUsers DESC LIMIT 100
use sampling
/* Raw-table */
CREATE TABLE StatsFull (
/* .. */
) ENGINE = MergeTree()
PARTITION BY toMonday(Date)
SAMPLE BY xxHash32(Uid) /* < -- */
ORDER BY Time, xxHash32(Uid)
/* Applying sampling to raw-table can make faster the short-term queries (period in several hours etc) */
SELECT Name, uniq(PcId) as cnt
FROM StatsFull
SAMPLE 0.05 /* <-- */
WHERE Time > subtractHours(now(), 6) /* <-- hours-period */
GROUP BY Name
ORDER BY cnt DESC LIMIT 100
/* Aggregated-table */
CREATE MATERIALIZED VIEW StatsAggregated (
Date Date,
ProductName String,
ErrorCode Int32,
Name String,
Version String,
UniqUsers AggregateFunction(uniq, String),
) ENGINE = AggregatingMergeTree()
PARTITION BY toMonday(Date)
SAMPLE BY intHash32(toInt32(Date)) /* < -- not sure that is good to choose */
ORDER BY (intHash32(toInt32(Date)), ProductName, ErrorCode, Name, Version)
SELECT /* .. */ FROM StatsFull GROUP BY /* .. */**
/* Applying sampling to aggregated-table can make faster the long-term queries (period in several weeks, months etc) */
SELECT Name
FROM StatsAggregated
SAMPLE 0.1 /* < -- */
WHERE Date > subtractMonths(toDate(now()), 3) /* <-- months-period */
GROUP BY Name
ORDER BY uniqMerge(UniqUsers) DESC LIMIT 100
use distributed query processing. Splitting data into several parts (shards) allows making distributed processing; additional increase of processing performance gives using distributed_group_by_no_merge-query setting.

Answer 2

if you need transpone array to rows you could use arrayJoin
SELECT Name, arrayJoin(topK(100)(Count)) AS top100_Count FROM Stats