Materialzed view works for few days and then stops - clickhouse

I have these three table (I cleaned them)
CREATE TABLE Record (
`visitId` String,
`visitorId` String,
`pageUrl` LowCardinality(String),
`createdAtDay` Date DEFAULT now()
) ENGINE = MergeTree PARTITION BY toYYYYMM(createdAtDay) PRIMARY KEY (
visitorId,
visitId,
pageUrl,
createdAtDay
)
ORDER BY
(visitorId, visitId, pageUrl)
CREATE MATERIALIZED VIEW DurationPerPage (
`visits` Int64 CODEC(DoubleDelta, LZ4),
`pageUrl` LowCardinality(String),
`visitors` Int64 CODEC(DoubleDelta, LZ4),
`duration` Int64 CODEC(DoubleDelta, LZ4),
`createdAtDay` Date,
) ENGINE = SummingMergeTree((visits, visitors, duration))
ORDER BY
(createdAtDay, pageUrl) AS
SELECT
countDistinct(visitId) AS visits,
cutQueryStringAndFragment(pageUrl) AS pageUrl,
countDistinct(visitorId) AS visitors,
sum(e.value) AS duration,
createdAtDay
FROM
Record AS r
LEFT JOIN Events AS e ON (r.visitId = e.visitId)
AND (e.eventType = 6)
WHERE
pageType LIKE '%single%'
GROUP BY
(createdAtDay, pageUrl);
CREATE TABLE Events (
`visitId` String,
`visitorId` String,
`value` Int64 CODEC(DoubleDelta, LZ4),
`eventType` Int16 CODEC(DoubleDelta, LZ4)
) ENGINE = MergeTree PARTITION BY (toYYYYMM(createdAtDay), eventType) PRIMARY KEY (visitId, eventType, createdAtDay)
ORDER BY
(visitId, eventType, createdAtDay)
as you can see I'm using both Record and Events table to feed my materialzed view. it works good for few days and then it stops and starts saving weird data (mostly zeros at the duration field) and I have then to delete and recreate it.
is there a related bug to this ? or something is wrong the View ?

Related

Clickhouse GraphiteMergeTree Table migrate from deprecated format_version

I tried 2 ways described here enter link description here
Edit metadata file
CREATE TABLE graphite.data_test
(
Path String,
Value Float64,
Time UInt32,
Date Date,
Timestamp UInt32
)
ENGINE = GraphiteMergeTree(Date, (Path, Time), 8192, 'graphite_rollup')
alter table graphite.data_test attach partition 202208 from graphite.data;
detach table graphite.data_test;
vi /var/lib/clickhouse/metadata/graphite/data_test.sql
ATTACH TABLE data_test
(
Path String,
Value Float64,
Time UInt32,
Date Date,
Timestamp UInt32
)
ENGINE = GraphiteMergeTree('graphite_rollup')
PARTITION BY toYYYYMM(Date)
ORDER BY (Path, Time)
SETTINGS index_granularity = 8192;
attach table graphite.data_test;
ERROR: MergeTree data format version on disk doesn't support custom partitioning.
Copy partitions
CREATE TABLE data_test
(
Path String,
Value Float64,
Time UInt32,
Date Date,
Timestamp UInt32
)
ENGINE = GraphiteMergeTree('graphite_rollup')
PARTITION BY toYYYYMM(Date)
ORDER BY (Path, Time)
SETTINGS index_granularity = 8192;
alter table graphite.data_test attach partition 202208 from graphite.data;
ERROR: Tables have different format_version.
Can you tell me if there is any workaround, a way to change the deprecate table to a new format?
it is possible only using console app and only for parts (not tables) and you need to build this app by yourself.
this app has different names (dependent on version) convert-parts-from-old-format / convert-month-partitioned-parts

clickhouse create table Exception: Aggregate function minState(origin_user) is found in wrong place in query

CREATE TABLE user_dwd.user_tag_bitmap_local
(
`tag` String,
`tag_item` String,
`p_day` Date,
`origin_user` UInt64,
`users` AggregateFunction(min, UInt64) MATERIALIZED minState(origin_user)
)
ENGINE = AggregatingMergeTree()
PARTITION BY toYYYYMMDD(p_day)
ORDER BY (tag, tag_item)
SETTINGS index_granularity = 8192;
when running sql to create table, show error:
[2021-10-17 12:05:28] Code: 184, e.displayText() = DB::Exception: Aggregate function minState(origin_user) is found in wrong place in query: While processing minState(origin_user) AS users_tmp_alter9508717652815860223: default expression and column type are incompatible. (version 21.8.4.51 (official build))
how to solve the error?
minState is an aggregating function, you cannot use it like this (it is for queries with a groupby section).
To solve it you can use MATERIALIZED initializeAggregation... or MATERIALIZED arrayReduce(minState...
But actually you don't need the second column.
You are looking for SimpleAggregateFunction:
https://clickhouse.com/docs/en/sql-reference/data-types/simpleaggregatefunction/
CREATE TABLE user_dwd.user_tag_bitmap_local
(
`tag` String,
`tag_item` String,
`p_day` Date,
`origin_user` SimpleAggregateFunction(min, UInt64) ---<<<-----
)
ENGINE = AggregatingMergeTree()
PARTITION BY toYYYYMMDD(p_day)
ORDER BY (tag, tag_item)
SETTINGS index_granularity = 8192;
https://clickhouse.com/docs/en/sql-reference/functions/other-functions/#initializeaggregation
CREATE TABLE user_tag_bitmap_local
(
`tag` String,
`tag_item` String,
`p_day` Date,
`origin_user` UInt64,
`users` AggregateFunction(min, UInt64) MATERIALIZED initializeAggregation('minState', origin_user)
)
ENGINE = AggregatingMergeTree
PARTITION BY toYYYYMMDD(p_day)
ORDER BY (tag, tag_item)
SETTINGS index_granularity = 8192
https://clickhouse.com/docs/en/sql-reference/functions/array-functions/#arrayreduce
CREATE TABLE user_tag_bitmap_local
(
`tag` String,
`tag_item` String,
`p_day` Date,
`origin_user` UInt64,
`users` AggregateFunction(min, UInt64) MATERIALIZED arrayReduce('minState', [origin_user])
)
ENGINE = AggregatingMergeTree
PARTITION BY toYYYYMMDD(p_day)
ORDER BY (tag, tag_item)
SETTINGS index_granularity = 8192

clickhouse MATERIALIZED VIEW issues

I created MATERIALIZED VIEW like this :
create target table:
CREATE TABLE user_deatils_daily (
day date,
hour UInt8 ,
appid UInt32,
isp String,
city String,
country String,
session_count UInt64,
avg_score AggregateFunction(avg, Float32),
min_revenue AggregateFunction(min, Float32),
max_load_time AggregateFunction(max, Int32)
)
ENGINE = SummingMergeTree()
PARTITION BY toRelativeWeekNum(day)
ORDER BY (day,hour)
create mv:
CREATE MATERIALIZED VIEW user_deatils_daily_mv
TO user_deatils_daily as
select toDate(session_ts) as day, toHour(toDateTime(session_ts)) as hour,appid,isp,city,country,
count(session_uuid) as session_count,avgState() as avg_score,
minState(revenue) as min_revenue,
maxState(perf_page_load_time) as max_load_time
from user_deatils where toDate(session_ts)>='2020-08-26' group by session_ts,appid,isp,city,country
the data in the target table starting to fill with data.
after some times the target table is getting fill with new data and doesn't' save the old one.
why is that?
SummingMergeTree() PARTITION BY toRelativeWeekNum(day) ORDER BY (day,hour)
means calculate sums groupby toRelativeWeekNum(day), day,hour)
user_deatils_daily knows nothing about user_deatils_daily_mv. They are not related.
user_deatils_daily_mv just does inserts into user_deatils_daily
SummingMergeTree knows nothing about group by session_ts,appid,isp,city,country
I would expect to see ORDER BY (ts,appid,isp,city,country);
I would do:
CREATE TABLE user_details_daily
( ts DateTime,
appid UInt32,
isp String,
city String,
country String,
session_count SimpleAggregateFunction(sum,UInt64),
avg_score AggregateFunction(avg, Float32),
min_revenue SimpleAggregateFunction(min, Float32),
max_load_time SimpleAggregateFunction(max, Int32) )
ENGINE = AggregatingMergeTree()
PARTITION BY toStartOfWeek(ts)
ORDER BY (ts,appid,isp,city,country);
CREATE MATERIALIZED VIEW user_deatils_daily_mv TO user_details_daily
as select
toStartOfHour(toDateTime(session_ts)) ts,
appid,
isp,
city,
country,
count(session_uuid) as session_count ,
avgState() as avg_score,
min(revenue) as min_revenue,
max(perf_page_load_time) as max_load_time
from user_details
where toDate(session_ts)>='2020-08-26' group by ts,appid,isp,city,country;

How to achieve recursive join in clickhouse?

I have three tables A, B and C. B has a fix balance, while A has amount which required to update with fix balance and push these combine records into C.
ex.-
B has 100rs balance, A has three records 50,-10,-40.
I am using a mview which does A LEFT JOIN B and updating records in C.
output required in C:
150,140,100
output comes in C:
150,90,60
since its a join its taking balance for all three records as 100 and that's the reason I am getting 150,90,60. But I want 150,140,100.
Is there any way to do it in clickhouse ?
schemas for A, B and C are
DROP TABLE IF EXISTS A;
CREATE TABLE A
(
id UInt64,
`date` DateTime,
user_id UInt64,
status LowCardinality(String),
amount Float64,
sign Int8
) ENGINE = CollapsingMergeTree(sign) order by (user_id, id);
DROP TABLE IF EXISTS C;
create table C
(
user_id UInt64,
id UInt64,
closing_balance Float64,
amount Float64,
status LowCardinality(String),
`date` DateTime,
sign Int8
) ENGINE = CollapsingMergeTree(sign) PARTITION BY toYYYYMM(date)
order by (user_id, id);
DROP TABLE IF EXISTS B;
create table B
(
user_id UInt64,
id UInt64,
closing_balance Float64,
amount Float64,
status LowCardinality(String),
`date` DateTime,
sign Int8
) ENGINE = CollapsingMergeTree(sign) order by (user_id);
DROP TABLE IF EXISTS B_mview;
CREATE MATERIALIZED VIEW B_mview to B
as SELECT * from C order by id asc;
insertion query is :
DROP TABLE IF EXISTS A_mview;
CREATE MATERIALIZED VIEW A_mview TO C AS SELECT A.company_id AS
company_id,
A.user_id AS user_id,
A.id AS id,
round((multiIf(A.status != 'Complete',
B.closing_balance,plus(A.amount,B.closing_balance))),2) AS
closing_balance,
A.amount AS amount,
A.date AS date,
toInt8(1) AS sign
FROM A LEFT JOIN B ON A.user_id=B.user_id

Clickhouse - cross table TTL expressions

Is it possible to define the TTL for a table in Clickhouse so that it references other table? Let's say I have a chat application and in my database I have two tables: chats and chat_messages. Chats have start and stop time information and I want to delete old chats along with their messages entirely when they expire - so basing on the chat stop_time. I tried to create those tables in following way:
db43af298bb9 :) CREATE TABLE chats (id Int64, start_time DateTime, stop_time DateTime) ENGINE = MergeTree() ORDER BY (start_time, id) TTL stop_time + INTERVAL 1 MONTH;
CREATE TABLE chats
(
`id` Int64,
`start_time` DateTime,
`stop_time` DateTime
)
ENGINE = MergeTree()
ORDER BY (start_time, id)
TTL stop_time + toIntervalMonth(1)
Ok.
0 rows in set. Elapsed: 0.014 sec.
db43af298bb9 :) CREATE TABLE chat_messages (id Int64, text String, chat_id Int64) ENGINE = MergeTree() ORDER BY id TTL (SELECT stop_time from chats where chats.id = chat_id) + INTERVAL 1 MONTH;
CREATE TABLE chat_messages
(
`id` Int64,
`text` String,
`chat_id` Int64
)
ENGINE = MergeTree()
ORDER BY id
TTL
(
SELECT stop_time
FROM chats
WHERE chats.id = chat_id
) + toIntervalMonth(1)
Received exception from server (version 19.16.10):
Code: 47. DB::Exception: Received from localhost:9000. DB::Exception: Missing columns: 'chat_id' while processing query: 'SELECT stop_time FROM chats WHERE id = chat_id', required columns: 'id' 'chat_id' 'stop_time', source columns: 'stop_time' 'id' 'start_time'.
0 rows in set. Elapsed: 0.017 sec.
The TTL definition for the second table fails because it tries to find the 'call_id' column in 'chats' table instead of the source 'chat_messages' table. Is what I'm trying to achieve even possible or am I forced to use ALTER DELETE mechanism instead?

Resources