I know that the distributed node does not combine intermediate results from shards by using distributed_group_by_no_merge.
The following SQL
select sum(xxxxx),xxxxx from (
select sum(xxxx),xxxx
from (
select count(xxx),xxx
from distributed_table group by xxx )
group by xxxx SETTINGS distributed_group_by_no_merge = 1
) group by xxxxx
I want to know that which part of sql will be sent to MergeTree node to execute by using distributed_group_by_no_merge？ is it？select count(xxx),xxx from distributed_table group by xxx ) group by xxxx SETTINGS distributed_group_by_no_merge = 1
how does the parameter of distributed_group_by_no_merge change the behavior of distributed query？which part of sql execute on MergeTree node and which part of sql execute on distributed node?
distributed_group_by_no_merge-param affects the way how the initiator-node (it is a node which runs distributed query) will form the final result of a distributed query:
either by merging aggregated intermediate states coming from shards by itself (it required copying full aggregated intermediate states from shards to initiator-node) [distributed_group_by_no_merge = 0 (default mode)]
or get already final result from shards (when each shard merges an intermediate aggregation state on its side and send to initiator-node only the final result). It provides a significant improvement in performance and resource consumption but requires the right selection of the sharding key [distributed_group_by_no_merge = 1]
I would put distributed_group_by_no_merge at the same level of subquery where defined distributed table to explicitly define your intention and avoid confusion when there are several distributed-subqueries.
Let's look at the way how to check the differences between the two modes (will use _shard_num-virtual column):
distributed_group_by_no_merge=0
SELECT
groupUniqArray(_shard_num) AS shards,
..
FROM table
WHERE ..
GROUP BY ..
SETTINGS distributed_group_by_no_merge = 0
/* Aggregated states were merged into ONE result set on initiator-node.
┌─shards────┬─ ..
│ [2, 1, 3] │ ..
└───────────┴─ ..
*/
distributed_group_by_no_merge=1
SELECT
groupUniqArray(_shard_num) AS shards,
..
FROM table
WHERE ..
GROUP BY ..
SETTINGS distributed_group_by_no_merge = 1
/* Get a set of final results (not aggregated states) from each shard. They should be unioned manually.
┌─shards─┬─ ..
│ [2] │ ..
│ [1] │ ..
│ [3] │ ..
└────────┴─ ..
*/
How to avoid merging high cardinality sub-select aggregations on distributed tables
Related
Documentation declares Atomicity for exchanging two tables, but what about multiple tables? Following query works well and perform exchanging within tables A <=> B and C <=> D:
EXCHANGE TABLES "default"."A" AND "default"."B", "default"."C" AND "default"."D"
Does EXCHANGE statement on Atomic engine provide atomic capabilities for multiple tables?
Can be situation when A <=> B exchanged but C <=> D not? Does atomicity spreads on whole EXCHANGE query?
They are not atomic.
insert into a.x1 select 1;
insert into a.x1 select 2;
exchange tables a.x1 and a.x2, a.y1 and a.y2;
Received exception from server (version 22.6.1):
Code: 521. DB::Exception: Received from localhost:9000.DB::ErrnoException: Paths cannot be exchanged because /var/lib/clickhouse/store/209/209474d2-5d64-4ca0-8b45-9abf4109235a/y1.sql or /var/lib/clickhouse/store/209/209474d2-5d64-4ca0-8b45-9abf4109235a/y2.sql does not exist, errno: 2, strerror: No such file or directory. (ATOMIC_RENAME_FAIL)
select * from a.x1;
Ok.
0 rows in set. Elapsed: 0.002 sec.
select * from a.x2
┌─A─┐
│ 2 │
└───┘
┌─A─┐
│ 1 │
└───┘
Atomicy is not about DML transactions. It's about visibility for selects. That queries do not see intermediate state and do not throw table . does not exists...
We've known materialized view is triggered by insertion. Large amount of data from one insertion will be divided into multiple blocks, each block will trigger one MV select, according to this doc.
Will MV select triggered after all the rows of a block received? Or it depends on the select? Say, if the select tries to count rows of inserted data, it won't be trigger until all data received? If the select is just doing data replication/reversed indexing, it will be triggered as soon as one record of the block received?
Insert triggers MatView after block is inserted into the main table. So the insert just passes the pointer to the block of rows (in memory) into MatView.
-- by default clickhouse-client forms blocks = 1048545
-- clickhouse-client by itself parses input stream and inserts into
-- server in Native format
select value from system.settings where name = 'max_insert_block_size';
1048545
-- here is a setup which traces sizes of blocks in rows
create table test(a Int, b String) engine=Null;
create materialized view test_mv
Engine=MergeTree order by ts as
select now() ts, count() rows from test;
-- create file with 100mil rows in TSV format
clickhouse-client -q "select number, 'x' from numbers(100000000) format TSV" >test.tsv
clickhouse-client -q "insert into test format TSV" <test.tsv
select max(rows), min(rows), count() from test_mv;
┌─max(rows)─┬─min(rows)─┬─count()─┐
│ 1947598 │ 32 │ 65 │
└───────────┴───────────┴─────────┘
-- 1947598 -<- several blocks were squashed into a single block because
-- of parallel parsing
-- 65 blocks were passed
truncate test_mv;
-- let's disable parallel parsing
clickhouse-client --input_format_parallel_parsing=0 -q "insert into test format TSV" <test.tsv
select max(rows), min(rows), count() from test_mv;
┌─max(rows)─┬─min(rows)─┬─count()─┐
│ 1048545 │ 388225 │ 96 │
└───────────┴───────────┴─────────┘
-- got 1048545 = max_insert_block_size
-- 96 blocks were passed
-- 100000000 - 95 * 1048545 = 388225
-- (95 blocks by max_insert_block_size and a remain 1 block = 388225 rows
truncate test_mv;
-- let's grow max_insert_block_size
clickhouse-client --max_insert_block_size=10000000000 --input_format_parallel_parsing=0 -q "insert into test format TSV" <test.tsv
select max(rows), min(rows), count() from test_mv;
┌─max(rows)─┬─min(rows)─┬─count()─┐
│ 100000000 │ 100000000 │ 1 │
└───────────┴───────────┴─────────┘
-- 1 block == 100 mil rows
more https://kb.altinity.com/altinity-kb-queries-and-syntax/atomic-insert/
Ive a Clickhouse query question, Im pretty new to Clickhouse so maybe its an easy one for the experts ;)! We have a single table with events in, each event is linked to a product fe product_click, product_view. I want to extract the data grouped by product but in a single line I need all types of events in a separated column so I can sort on it.
I already wrote this query:
SELECT product_id,
arrayMap((x, y) -> (x, y),
(arrayReduce('sumMap', [(groupArrayArray([event_type]) as arr)],
[arrayResize(CAST([], 'Array(UInt64)'), length(arr), toUInt64(1))]) as s).1, s.2) events
FROM events
GROUP BY product_id
Result:
┌─────────────────────────product_id───┬─events─────────────────────────────────────────────────────────────────────────────────────┐
│ 0071f1e4-a484-448e-8355-64e2fea98fd5 │ [('PRODUCT_CLICK',1341),('PRODUCT_VIEW',11)] │
│ 406f4707-6bad-4d3f-9544-c74fdeb1e09d │ [('PRODUCT_CLICK',1),('PRODUCT_VIEW',122),('PRODUCT_BUY',37)] │
│ 94566b6d-6e23-4264-ad76-697ffcfe60c4 │ [('PRODUCT_CLICK',1027),('PRODUCT_VIEW',7)] │
...
Is there any way to convert to arrayMap to columns with a sort key?
So we can filter on the most clicked products first, or the most viewed?
Another question, is having this kind of queries a good idea to always execute, or should we create a MATERIALIZED view for it?
Thanks!
SQL does not allow variable number of columns.
the only way for you
SELECT product_id,
countIf(event_type = 'PRODUCT_CLICK') PRODUCT_CLICK,
countIf(event_type = 'PRODUCT_VIEW') PRODUCT_VIEW,
countIf(event_type = 'PRODUCT_BUY') PRODUCT_BUY
FROM events
GROUP BY product_id
I got String column uin in several tables, how do I can effectively join on uin these tables?
In Vertica database we use hash(uin) to transform string column into hash with Int data type - it significantly boosts efficiency in joins - could you recommend something like this? I tried CRC32(s) but it seems to work wrong.
At this moment the CH not very good cope with multi-joins queries (DB star-schema) and the query optimizer not good enough to rely on it completely.
So it needs to explicitly say how to 'execute' a query by using subqueries instead of joins.
Let's emulate your query:
SELECT table_01.number AS r
FROM numbers(87654321) AS table_01
INNER JOIN numbers(7654321) AS table_02 ON (table_01.number = table_02.number)
INNER JOIN numbers(654321) AS table_03 ON (table_02.number = table_03.number)
INNER JOIN numbers(54321) AS table_04 ON (table_03.number = table_04.number)
ORDER BY r DESC
LIMIT 8;
/*
┌─────r─┐
│ 54320 │
│ 54319 │
│ 54318 │
│ 54317 │
│ 54316 │
│ 54315 │
│ 54314 │
│ 54313 │
└───────┘
8 rows in set. Elapsed: 4.244 sec. Processed 96.06 million rows, 768.52 MB (22.64 million rows/s., 181.10 MB/s.)
*/
On my PC it takes ~4 secs. Let's rewrite it using subqueries to significantly speed it up.
SELECT number AS r
FROM numbers(87654321)
WHERE number IN (
SELECT number
FROM numbers(7654321)
WHERE number IN (
SELECT number
FROM numbers(654321)
WHERE number IN (
SELECT number
FROM numbers(54321)
)
)
)
ORDER BY r DESC
LIMIT 8;
/*
┌─────r─┐
│ 54320 │
│ 54319 │
│ 54318 │
│ 54317 │
│ 54316 │
│ 54315 │
│ 54314 │
│ 54313 │
└───────┘
8 rows in set. Elapsed: 0.411 sec. Processed 96.06 million rows, 768.52 MB (233.50 million rows/s., 1.87 GB/s.)
*/
There are other ways to optimize JOIN:
use External dictionary to get rid of join on 'small'-table
use Join table engine
use ANY-strictness
use specific settings like join_algorithm, partial_merge_join_optimizations etc
Some useful refs:
Altinity webinar: Tips and tricks every ClickHouse user should know
Altinity webinar: Secrets of ClickHouse Query Performance
Answer update:
To less storage consumption for String-column consider changing column type to LowCardinality (link 2) that significantly decrease the size of a column with many duplicated elements.
Use this query to get the size of columns:
SELECT
name AS column_name,
formatReadableSize(data_compressed_bytes) AS data_size,
formatReadableSize(marks_bytes) AS index_size,
type,
compression_codec
FROM system.columns
WHERE database = 'db_name' AND table = 'table_name'
ORDER BY data_compressed_bytes DESC
To get a numeric representation of a string need to use one of hash-functions.
SELECT 'jsfhuhsdf', xxHash32('jsfhuhsdf'), cityHash64('jsfhuhsdf');
I have 3 tables events_0, events_1, events_2 with Engine = MergeTree
and 1 table events with Engine = Merge
CREATE TABLE events as events_0 ENGINE=Merge(currentDatabase(), '^events');
When I run sql query like
select uuid from events where uuid = 'XXXX-YYY-ZZZZ';
I've got a duplicated response
┌─uuid──────────┐
│ XXXX-YYY-ZZZZ │
└───────────────┘
┌─uuid──────────┐
│ XXXX-YYY-ZZZZ │
└───────────────┘
Try adding _table to the select clause to see which table was generating the data.
select _table, uuid from events where uuid = 'XXXX-YYY-ZZZZ';
It looks like a self-recursion to me. You might need to rename the merge table that won't be matched by regex ^events