ClickHouse MergeTree slow SELECT with ORDER BY - clickhouse

Question

I'm starting to learn CH and seem to be running into dead ends while trying to improve my query speed, the table is created like this
CREATE TABLE default.stats(
aa String,
ab String,
user_id UInt16,
ac UInt32,
ad UInt8,
ae UInt8,
created_time DateTime,
created_date Date,
af UInt8,
ag UInt32,
ah UInt32,
ai String,
aj String)
ENGINE = MergeTree
PARTITION BY toYYYYMM(created_time)
ORDER BY(created_time, user_id)
and I'm running a query like so
SELECT ad, created_time, ab, aa, user_id, ac, ag, af
FROM stats
WHERE user_id = 1 AND lowerUTF8(ab) = 'xxxxxxxxx' AND ad != 12
ORDER BY created_time DESC
LIMIT 50 OFFSET 0
this is the result 50 rows in set. Elapsed: 2.881 sec. Processed 74.62 million rows
and if I run the same query without the order part, 50 rows in set. Elapsed: 0.020 sec. Processed 49.15 thousand rows
Why does it seem to process all the rows in the table if in theory the query only has to order around 10k(all the rows returned without the limit) rows? What am I missing and/or how could I improve the speed of CH?

Answer 1

try ORDER BY created_time DESC, user_id
optimize_read_in_order feature were implemented in ClickHouse release 19.14.3.3, 2019-09-10

Answer 2

CH 19.17.4.11
CREATE TABLE stats
(
`aa` String,
`ab` String,
`user_id` UInt16,
`ac` UInt32,
`ad` UInt8,
`ae` UInt8,
`created_time` DateTime,
`created_date` Date,
`af` UInt8,
`ag` UInt32,
`ah` UInt32,
`ai` String,
`aj` String
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(created_time)
ORDER BY (created_time, user_id)
insert into stats(created_time, user_id) select toDateTime(intDiv(number,100)), number%103 from numbers(100000000)
SELECT ad, created_time, ab, aa, user_id, ac, ag, af
FROM stats
ORDER BY created_time DESC
LIMIT 5 OFFSET 0
5 rows in set. Elapsed: 0.013 sec. Processed 835.84 thousand rows,
set optimize_read_in_order = 0
SELECT ad, created_time, ab, aa, user_id, ac, ag, af
FROM stats
ORDER BY created_time DESC
LIMIT 5 OFFSET 0
5 rows in set. Elapsed: 0.263 sec. Processed 100.00 million rows
Check a difference
set optimize_read_in_order = 0 VS set optimize_read_in_order = 1
I don't understand why optimize_read_in_order is not working in your case.