I am trying to check clickhous database(server) active connections from console.
In MYSQL there are many commands Like:
SHOW STATUS WHERE `variable_name` = 'Threads_connected';
show status like 'Threads_connected';
SHOW VARIABLES LIKE "max_connections";
It will return result like this below format.
'Threads_connected', '2'
'max_connections', '151'
As well as we can from mysql workbench server status menu.
I am checking with below query, but it is not returning any appropriate result.
SELECT * FROM system.metrics WHERE metric LIKE '%Connection'
┌─metric────────────────┬─value─┬
│ TCPConnection │ 1 │
│ MySQLConnection │ 0 │
│ HTTPConnection │ 0 │
│ InterserverConnection │ 0 │
│ PostgreSQLConnection │ 0 │
SELECT * FROM system.events LIMIT 1
Row 1:
──────
event: Query
value: 30
description: Number of queries....
SELECT * FROM system.settings WHERE name LIKE 'conn%' LIMIT 1
Row 1:
──────
name: connect_timeout
value: 10
changed: 0
description: Connection timeout if there are no replicas.
min: ᴺᵁᴸᴸ
max: ᴺᵁᴸᴸ
readonly: 0
type: Seconds
Please share some idea how can check open session or connection.
Thank you,
Bimal.
The query
SELECT * FROM system.metrics WHERE metric LIKE '%Connection'
is a right query, but look like not all metrics currently calculate properly. You could create an issue on github about it.
Related
I had executed the following query but it has processed ~1B rows and took total time of 75 seconds for a simple count.
SELECT count(*)
FROM events_distributed
WHERE (orgId = '174a4727-1116-4c5c-8234-ab76f2406c4a') AND (timestamp >= '2022-12-05 00:00:00.000000000')
Query id: e4312ff5-6add-4757-8deb-d68e0f3e29d9
┌──count()─┐
│ 13071204 │
└──────────┘
1 row in set. Elapsed: 74.951 sec. Processed 979.00 million rows, 8.26 GB (13.06 million rows/s., 110.16 MB/s.)
I am wondering how I can speed this up? My events table has the following partition by and order by columns and a bloom filter index on orgid
PARTITION BY toDate(timestamp)
ORDER BY (timestamp);
INDEX idx_orgid orgid TYPE bloom_filter(0.01) GRANULARITY 1,
Below is the execution plan
EXPLAIN indexes = 1
SELECT count(*)
FROM events_distributed
WHERE (orgid = '174a4727-1116-4c5c-8234-ab76f240fc4a') AND (timestamp >= '2022-12-05 00:00:00.000000000') AND (timestamp <= '2022-12-06 00:00:00.000000000')
Query id: 879c2ce5-c4c7-4efc-b0e2-25613848afad
┌─explain────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expression ((Projection + Before ORDER BY)) │
│ MergingAggregated │
│ Union │
│ Aggregating │
│ Expression (Before GROUP BY) │
│ Filter (WHERE) │
│ ReadFromMergeTree (users.events) │
│ Indexes: │
│ MinMax │
│ Keys: │
│ timestamp │
│ Condition: and((timestamp in (-Inf, '1670284800']), (timestamp in ['1670198400', +Inf))) │
│ Parts: 12/342 │
│ Granules: 42122/407615 │
│ Partition │
│ Keys: │
│ toDate(timestamp) │
│ Condition: and((toDate(timestamp) in (-Inf, 19332]), (toDate(timestamp) in [19331, +Inf))) │
│ Parts: 12/12 │
│ Granules: 42122/42122 │
│ PrimaryKey │
│ Keys: │
│ timestamp │
│ Condition: and((timestamp in (-Inf, '1670284800']), (timestamp in ['1670198400', +Inf))) │
│ Parts: 12/12 │
│ Granules: 30696/42122 │
│ Skip │
│ Name: idx_orgid │
│ Description: bloom_filter GRANULARITY 1 │
│ Parts: 8/12 │
│ Granules: 20556/30696 │
│ ReadFromRemote (Read from remote replica) │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
32 rows in set. Elapsed: 0.129 sec.
How can I speed up this query? because processing 1B rows to give a count of 13M sounds like something is total off. Does creating a SET index on orgid any better? because I will have a max of 10K orgs
The queries I typicall run are
SELECT org_level, min(timestamp) as minTimeStamp,max(timestamp) as maxTimeStamp, toStartOfInterval(toDateTime(timestamp), INTERVAL <step> second) as roundedDownTs, count(*) as cnt, orgid
FROM events_distributed
WHERE orgid = 'foo' and timestamp BETWEEN <one week>
GROUP BY roundedDownTs, orgid, org_level
ORDER BY roundedDownTs DESC;
please note <step> here would be any of the following values 0, 60, 240, 1440, 10080
and another query for a one week time slice but it can be any time slice and always want the results in descending order because of timeseries
SELECT org_text
FROM events_distributed
WHERE (orgid = '174a4727-1116-4c5c-8234-ab76f2406c4a') AND (timestamp >= '2022-12-01 00:00:00.000000000' and timestamp <= '2022-12-07 00:00:00.000000000') order by timestamp DESC LIMIT 51;
You don't use primary index
I suggest is to use
PARTITION BY toDate(timestamp)
ORDER BY (orgId, timestamp)
https://kb.altinity.com/engines/mergetree-table-engine-family/pick-keys/
And remove bloom_filter index.
I want to check if a Map contains all the entries I need.
> create table map_test (my_map Map(String, String)) engine = Memory;
> insert into map_test values ({'k1': 'v1', 'k2': 'v2'}), ({'k1': 'v1', 'k2': 'v2', 'k3': 'v3'}), ({'k1': 'v1', 'k4': 'v4'});
> select * from map_test;
┌─my_map──────────────────────────┐
│ {'k1':'v1','k2':'v2'} │
│ {'k1':'v1','k2':'v2','k3':'v3'} │
│ {'k1':'v1','k4':'v4'} │
└─────────────────────────────────┘
3 rows in set. Elapsed: 0.001 sec.
-- get the rows that "my_map" contains all entries I need.
> select * from map_test where my_map['k1'] = 'v1' and my_map['k2'] = 'v2'; -- The SQL will be very long.
┌─my_map──────────────────────────┐
│ {'k1':'v1','k2':'v2'} │
│ {'k1':'v1','k2':'v2','k3':'v3'} │
└─────────────────────────────────┘
2 rows in set. Elapsed: 0.001 sec.
If I have a lot of entries in where clause, the SQL will be very long.
Is there a way to do it like the following SQL.
select * from map_test where mapContainsAll(my_map, {'k1': 'v1', 'k2': 'v2'});
I've read documentation about Map, but can't find a function like mapContainsAll.
Let's say I have a table defined as
CREATE TABLE orders (
sqlId Int64, -- orders.id from PSQL
isApproved UInt8, -- Boolean
comment String,
price Decimal(10, 2),
createdAt DateTime64(9, 'UTC'),
updatedAt DateTime64(9, 'UTC') DEFAULT NOW()
)
ENGINE = MergeTree
ORDER BY (createdAt, sqlId)
so two fields that might be changed in the source PSQL database - isApproved and comment
Naturally, if I sink some records from an MQ topic into it, I will end up with something like this
SELECT *
FROM orders
Query id: 50cd95a4-e581-41b5-82a4-7ec86771e4e5
┌─sqlId─┬─isApproved─┬──price─┬─comment───┬─────────────────────createdAt─┬─────────────────────updatedAt─┐
│ 1 │ 1 │ 100.00 │ some note │ 2021-11-08 16:24:07.000000000 │ 2021-11-08 16:27:29.000000000 │
└───────┴────────────┴────────┴───────────┴───────────────────────────────┴───────────────────────────────┘
┌─sqlId─┬─isApproved─┬──price─┬─comment─┬─────────────────────createdAt─┬─────────────────────updatedAt─┐
│ 1 │ 1 │ 100.00 │ │ 2021-11-08 16:24:07.000000000 │ 2021-11-08 16:27:22.000000000 │
└───────┴────────────┴────────┴─────────┴───────────────────────────────┴───────────────────────────────┘
┌─sqlId─┬─isApproved─┬──price─┬─comment─┬─────────────────────createdAt─┬─────────────────────updatedAt─┐
│ 1 │ 0 │ 100.00 │ │ 2021-11-08 16:24:07.000000000 │ 2021-11-08 16:27:17.000000000 │
└───────┴────────────┴────────┴─────────┴───────────────────────────────┴───────────────────────────────┘
In other words, a particular order was created first as a non-approved, then it was approved, then some comment was added.
Let's say I want to create a view that represents the total volume of orders per day.
A naive approach might be:
CREATE MATERIALIZED VIEW orders_volume_per_day (
day DateTime64(9, 'UTC'),
volume SimpleAggregateFunction(sum, Decimal(38, 2))
)
ENGINE = AggregatingMergeTree
PARTITION BY toYYYYMM(day)
ORDER BY day
AS SELECT
toStartOfDay(createdAt) as day,
sum(price) as volume
FROM orders
GROUP BY day
ORDER BY day ASC
However, it will be using all three redundant records, while I only need to use the latest one.
In my particular example, the view will return 300 (3x100) instead of just 100.
Is there any way to achieve the desired behavior in Clickhouse? I know that I can utilize VersionedCollapsingMergeTree somehow with the sign or version columns, but it seems that tools like clickhouse_sinker or snuba will not support it.
Assuming following schema:
CREATE TABLE test
(
date Date,
user_id UInt32,
user_answer UInt8,
user_multi_choice_answer Array(UInt8),
events UInt32
)
ENGINE = MergeTree() ORDER BY date;
And contents:
INSERT INTO test VALUES
('2020-01-01', 1, 5, [2, 3], 15),
('2020-01-01', 2, 6, [1, 2], 7);
Let's say I want to make a query "give me # of users and # of their events grouped by date and user_answer, with subtotals". That's easy:
select date, user_answer, count(distinct user_id), sum(events) from test group by date, user_answer with rollup;
┌───────date─┬─user_answer─┬─uniqExact(user_id)─┬─sum(events)─┐
│ 2020-01-01 │ 5 │ 1 │ 15 │
│ 2020-01-01 │ 6 │ 1 │ 7 │
│ 2020-01-01 │ 0 │ 2 │ 22 │
│ 0000-00-00 │ 0 │ 2 │ 22 │
└────────────┴─────────────┴────────────────────┴─────────────┘
What I can't easily do is making queries with overlapping groups, like when grouping by invidivual options of multiple choice question. For example:
# of users and # of their events grouped by date and user_multi_choice_answer, with subtotals
# of users and # of their events grouped by arbitrary hand-written grouping conditions, like "compare users with user_answer=5 and has(user_multi_choice_answer, 1) to users with has(user_multi_choice_answer, 2)"
For example, with the first query, I would like to see the following:
┌───────date─┬─user_multi_choice_answer─┬─uniqExact(user_id)─┬─sum(events)─┐
│ 2020-01-01 │ 1 │ 1 │ 15 │
│ 2020-01-01 │ 2 │ 2 │ 22 │
│ 2020-01-01 │ 3 │ 1 │ 7 │
│ 2020-01-01 │ 0 │ 2 │ 22 │
│ 0000-00-00 │ 0 │ 2 │ 22 │
└────────────┴──────────────────────────┴────────────────────┴─────────────┘
And for the second:
┌─my_grouping_id─┬─uniqExact(user_id)─┬─sum(events)─┐
│ 1 │ 1 │ 15 │ # users fulfilling arbitrary condition #1
│ 2 │ 2 │ 22 │ # users fulfilling arbitrary condition #2
│ 0 │ 2 │ 22 │ # subtotal
└────────────────┴────────────────────┴─────────────┘
The closest I can get to that is by using arrayJoin():
select date, arrayJoin(user_multi_choice_answer) as multi_answer, count(distinct user_id), sum(events)
from test group by date, multi_answer with rollup;
select arrayJoin(
arrayConcat(
if(user_answer=5 and has(user_multi_choice_answer, 3), [1], []),
if(has(user_multi_choice_answer, 2), [2], [])
)
) as my_grouping_id, count(distinct user_id), sum(events)
from test group by my_grouping_id with rollup;
But that's not a good solution for two reasons:
While it calculates correct results for grouping, the result for sum(events) is not correct for subtotals (as duplicated rows count multiple times)
It doesn't seem efficient, as it makes a lot of data duplication (while I just want the same row to get aggregated into several groups)
So, again, I'm looking for a way that would allow me to easily make grouping of answers to multiple choice questions and gropings by arbitrary conditions on some columns. I'm okay with changing the schema to make that possible, but I'm mostly hoping Clickhouse has a built-in way to achieve that.
While it calculates correct results for grouping, the result for sum(events) is not correct for subtotals (as duplicated rows count multiple times)
You can manually create my_grouping_id = 0 without using rollup. For example,
select arrayJoin(
arrayConcat(
[0],
if(user_answer=5 and has(user_multi_choice_answer, 3), [1], []),
if(has(user_multi_choice_answer, 2), [2], [])
)
) as my_grouping_id, count(distinct user_id), sum(events)
from test group by my_grouping_id
It doesn't seem efficient, as it makes a lot of data duplication (while I just want the same row to get aggregated into several groups)
Currently it's not possible. But I see possibilities. I'll try to make a POC of GROUP BY ARRAY. It seems to be a valid use case.
I'm playing with data in csv format from https://dev.maxmind.com/geoip/geoip2/geolite2/.
Generally, it's data that map from ip block to asn and country.
I have 2 table both are Memory engine, first has 299727 records, second has 406685.
SELECT *
FROM __ip_block_to_country
LIMIT 5
┌─network────┬───────id─┬───min_ip─┬───max_ip─┬─geoname_id─┬─country_iso_code─┬─country_name─┐
│ 1.0.0.0/24 │ 16777216 │ 16777217 │ 16777472 │ 2077456 │ AU │ Australia │
│ 1.0.1.0/24 │ 16777472 │ 16777473 │ 16777728 │ 1814991 │ CN │ China │
│ 1.0.2.0/23 │ 16777728 │ 16777729 │ 16778240 │ 1814991 │ CN │ China │
│ 1.0.4.0/22 │ 16778240 │ 16778241 │ 16779264 │ 2077456 │ AU │ Australia │
│ 1.0.8.0/21 │ 16779264 │ 16779265 │ 16781312 │ 1814991 │ CN │ China │
└────────────┴──────────┴──────────┴──────────┴────────────┴──────────────────┴──────────────┘
SELECT *
FROM __ip_block_to_asn
LIMIT 5
┌─network──────┬─autonomous_system_number─┬─autonomous_system_organization─┬───────id─┬─subnet_count─┬───min_ip─┬───max_ip─┐
│ 1.0.0.0/24 │ 13335 │ Cloudflare Inc │ 16777216 │ 255 │ 16777217 │ 16777472 │
│ 1.0.4.0/22 │ 56203 │ Gtelecom-AUSTRALIA │ 16778240 │ 1023 │ 16778241 │ 16779264 │
│ 1.0.16.0/24 │ 2519 │ ARTERIA Networks Corporation │ 16781312 │ 255 │ 16781313 │ 16781568 │
│ 1.0.64.0/18 │ 18144 │ Energia Communications,Inc. │ 16793600 │ 16383 │ 16793601 │ 16809984 │
│ 1.0.128.0/17 │ 23969 │ TOT Public Company Limited │ 16809984 │ 32767 │ 16809985 │ 16842752 │
└──────────────┴──────────────────────────┴────────────────────────────────┴──────────┴──────────────┴──────────┴──────────┘
Now, i want to exam which country that covers entire ip pool of one asn. The below query is just to obtain index of statisfied country.
SELECT idx from(
SELECT
(
SELECT groupArray(min_ip),groupArray(max_ip),groupArray(country_iso_code),groupArray(country_name)
FROM __ip_block_to_country
) t,
arrayFilter((i,mii, mai) -> min_ip >= mii and max_ip <= mai, arrayEnumerate(t.1), t.1, t.2) as idx
FROM __ip_block_to_asn
);
I got following exception:
Received exception from server (version 1.1.54394):
Code: 241. DB::Exception: Received from localhost:9000, ::1. DB::Exception: Memory limit (for query) exceeded: would use 512.02 GiB (attempt to allocate chunk of 549755813888 bytes), maximum: 37.25 GiB.
My question is:
It seems like the statement SELECT groupArray(min_ip),groupArray(max_ip),groupArray(country_iso_code),groupArray(country_name) is executed along with every record of __ip_block_to_asn, then query needs so much memory. Is that true to my query ?
Scalar subquery is executed only once.
But to execute arrayFilter, arrays are multiplied by number of rows of processed blocks from __ip_block_to_asn table. It is something like cross join of two tables.
To overcome this, you can use smaller block size for SELECT from __ip_block_to_asn.
It is controlled by max_block_size setting. But for Memory tables, blocks always have the same size as when they was inserted into a table, regardless to max_block_size setting during SELECT. To allow flexible block size, you can reload this table to TinyLog engine.
CREATE TABLE __ip_block_to_asn2 ENGINE = TinyLog AS SELECT * FROM __ip_block_to_asn
Then execute:
SET max_block_size = 10;
SELECT idx from(
SELECT
(
SELECT groupArray(min_ip),groupArray(max_ip),groupArray(country_iso_code),groupArray(country_name)
FROM __ip_block_to_country
) t,
arrayFilter((i,mii, mai) -> min_ip >= mii and max_ip <= mai, arrayEnumerate(t.1), t.1, t.2) as idx
FROM __ip_block_to_asn2
);