I have three tables A, B and C. B has a fix balance, while A has amount which required to update with fix balance and push these combine records into C.
ex.-
B has 100rs balance, A has three records 50,-10,-40.
I am using a mview which does A LEFT JOIN B and updating records in C.
output required in C:
150,140,100
output comes in C:
150,90,60
since its a join its taking balance for all three records as 100 and that's the reason I am getting 150,90,60. But I want 150,140,100.
Is there any way to do it in clickhouse ?
schemas for A, B and C are
DROP TABLE IF EXISTS A;
CREATE TABLE A
(
id UInt64,
`date` DateTime,
user_id UInt64,
status LowCardinality(String),
amount Float64,
sign Int8
) ENGINE = CollapsingMergeTree(sign) order by (user_id, id);
DROP TABLE IF EXISTS C;
create table C
(
user_id UInt64,
id UInt64,
closing_balance Float64,
amount Float64,
status LowCardinality(String),
`date` DateTime,
sign Int8
) ENGINE = CollapsingMergeTree(sign) PARTITION BY toYYYYMM(date)
order by (user_id, id);
DROP TABLE IF EXISTS B;
create table B
(
user_id UInt64,
id UInt64,
closing_balance Float64,
amount Float64,
status LowCardinality(String),
`date` DateTime,
sign Int8
) ENGINE = CollapsingMergeTree(sign) order by (user_id);
DROP TABLE IF EXISTS B_mview;
CREATE MATERIALIZED VIEW B_mview to B
as SELECT * from C order by id asc;
insertion query is :
DROP TABLE IF EXISTS A_mview;
CREATE MATERIALIZED VIEW A_mview TO C AS SELECT A.company_id AS
company_id,
A.user_id AS user_id,
A.id AS id,
round((multiIf(A.status != 'Complete',
B.closing_balance,plus(A.amount,B.closing_balance))),2) AS
closing_balance,
A.amount AS amount,
A.date AS date,
toInt8(1) AS sign
FROM A LEFT JOIN B ON A.user_id=B.user_id
Related
I have these three table (I cleaned them)
CREATE TABLE Record (
`visitId` String,
`visitorId` String,
`pageUrl` LowCardinality(String),
`createdAtDay` Date DEFAULT now()
) ENGINE = MergeTree PARTITION BY toYYYYMM(createdAtDay) PRIMARY KEY (
visitorId,
visitId,
pageUrl,
createdAtDay
)
ORDER BY
(visitorId, visitId, pageUrl)
CREATE MATERIALIZED VIEW DurationPerPage (
`visits` Int64 CODEC(DoubleDelta, LZ4),
`pageUrl` LowCardinality(String),
`visitors` Int64 CODEC(DoubleDelta, LZ4),
`duration` Int64 CODEC(DoubleDelta, LZ4),
`createdAtDay` Date,
) ENGINE = SummingMergeTree((visits, visitors, duration))
ORDER BY
(createdAtDay, pageUrl) AS
SELECT
countDistinct(visitId) AS visits,
cutQueryStringAndFragment(pageUrl) AS pageUrl,
countDistinct(visitorId) AS visitors,
sum(e.value) AS duration,
createdAtDay
FROM
Record AS r
LEFT JOIN Events AS e ON (r.visitId = e.visitId)
AND (e.eventType = 6)
WHERE
pageType LIKE '%single%'
GROUP BY
(createdAtDay, pageUrl);
CREATE TABLE Events (
`visitId` String,
`visitorId` String,
`value` Int64 CODEC(DoubleDelta, LZ4),
`eventType` Int16 CODEC(DoubleDelta, LZ4)
) ENGINE = MergeTree PARTITION BY (toYYYYMM(createdAtDay), eventType) PRIMARY KEY (visitId, eventType, createdAtDay)
ORDER BY
(visitId, eventType, createdAtDay)
as you can see I'm using both Record and Events table to feed my materialzed view. it works good for few days and then it stops and starts saving weird data (mostly zeros at the duration field) and I have then to delete and recreate it.
is there a related bug to this ? or something is wrong the View ?
CREATE TABLE user_dwd.user_tag_bitmap_local
(
`tag` String,
`tag_item` String,
`p_day` Date,
`origin_user` UInt64,
`users` AggregateFunction(min, UInt64) MATERIALIZED minState(origin_user)
)
ENGINE = AggregatingMergeTree()
PARTITION BY toYYYYMMDD(p_day)
ORDER BY (tag, tag_item)
SETTINGS index_granularity = 8192;
when running sql to create table, show error:
[2021-10-17 12:05:28] Code: 184, e.displayText() = DB::Exception: Aggregate function minState(origin_user) is found in wrong place in query: While processing minState(origin_user) AS users_tmp_alter9508717652815860223: default expression and column type are incompatible. (version 21.8.4.51 (official build))
how to solve the error?
minState is an aggregating function, you cannot use it like this (it is for queries with a groupby section).
To solve it you can use MATERIALIZED initializeAggregation... or MATERIALIZED arrayReduce(minState...
But actually you don't need the second column.
You are looking for SimpleAggregateFunction:
https://clickhouse.com/docs/en/sql-reference/data-types/simpleaggregatefunction/
CREATE TABLE user_dwd.user_tag_bitmap_local
(
`tag` String,
`tag_item` String,
`p_day` Date,
`origin_user` SimpleAggregateFunction(min, UInt64) ---<<<-----
)
ENGINE = AggregatingMergeTree()
PARTITION BY toYYYYMMDD(p_day)
ORDER BY (tag, tag_item)
SETTINGS index_granularity = 8192;
https://clickhouse.com/docs/en/sql-reference/functions/other-functions/#initializeaggregation
CREATE TABLE user_tag_bitmap_local
(
`tag` String,
`tag_item` String,
`p_day` Date,
`origin_user` UInt64,
`users` AggregateFunction(min, UInt64) MATERIALIZED initializeAggregation('minState', origin_user)
)
ENGINE = AggregatingMergeTree
PARTITION BY toYYYYMMDD(p_day)
ORDER BY (tag, tag_item)
SETTINGS index_granularity = 8192
https://clickhouse.com/docs/en/sql-reference/functions/array-functions/#arrayreduce
CREATE TABLE user_tag_bitmap_local
(
`tag` String,
`tag_item` String,
`p_day` Date,
`origin_user` UInt64,
`users` AggregateFunction(min, UInt64) MATERIALIZED arrayReduce('minState', [origin_user])
)
ENGINE = AggregatingMergeTree
PARTITION BY toYYYYMMDD(p_day)
ORDER BY (tag, tag_item)
SETTINGS index_granularity = 8192
I created MATERIALIZED VIEW like this :
create target table:
CREATE TABLE user_deatils_daily (
day date,
hour UInt8 ,
appid UInt32,
isp String,
city String,
country String,
session_count UInt64,
avg_score AggregateFunction(avg, Float32),
min_revenue AggregateFunction(min, Float32),
max_load_time AggregateFunction(max, Int32)
)
ENGINE = SummingMergeTree()
PARTITION BY toRelativeWeekNum(day)
ORDER BY (day,hour)
create mv:
CREATE MATERIALIZED VIEW user_deatils_daily_mv
TO user_deatils_daily as
select toDate(session_ts) as day, toHour(toDateTime(session_ts)) as hour,appid,isp,city,country,
count(session_uuid) as session_count,avgState() as avg_score,
minState(revenue) as min_revenue,
maxState(perf_page_load_time) as max_load_time
from user_deatils where toDate(session_ts)>='2020-08-26' group by session_ts,appid,isp,city,country
the data in the target table starting to fill with data.
after some times the target table is getting fill with new data and doesn't' save the old one.
why is that?
SummingMergeTree() PARTITION BY toRelativeWeekNum(day) ORDER BY (day,hour)
means calculate sums groupby toRelativeWeekNum(day), day,hour)
user_deatils_daily knows nothing about user_deatils_daily_mv. They are not related.
user_deatils_daily_mv just does inserts into user_deatils_daily
SummingMergeTree knows nothing about group by session_ts,appid,isp,city,country
I would expect to see ORDER BY (ts,appid,isp,city,country);
I would do:
CREATE TABLE user_details_daily
( ts DateTime,
appid UInt32,
isp String,
city String,
country String,
session_count SimpleAggregateFunction(sum,UInt64),
avg_score AggregateFunction(avg, Float32),
min_revenue SimpleAggregateFunction(min, Float32),
max_load_time SimpleAggregateFunction(max, Int32) )
ENGINE = AggregatingMergeTree()
PARTITION BY toStartOfWeek(ts)
ORDER BY (ts,appid,isp,city,country);
CREATE MATERIALIZED VIEW user_deatils_daily_mv TO user_details_daily
as select
toStartOfHour(toDateTime(session_ts)) ts,
appid,
isp,
city,
country,
count(session_uuid) as session_count ,
avgState() as avg_score,
min(revenue) as min_revenue,
max(perf_page_load_time) as max_load_time
from user_details
where toDate(session_ts)>='2020-08-26' group by ts,appid,isp,city,country;
Is it possible to define the TTL for a table in Clickhouse so that it references other table? Let's say I have a chat application and in my database I have two tables: chats and chat_messages. Chats have start and stop time information and I want to delete old chats along with their messages entirely when they expire - so basing on the chat stop_time. I tried to create those tables in following way:
db43af298bb9 :) CREATE TABLE chats (id Int64, start_time DateTime, stop_time DateTime) ENGINE = MergeTree() ORDER BY (start_time, id) TTL stop_time + INTERVAL 1 MONTH;
CREATE TABLE chats
(
`id` Int64,
`start_time` DateTime,
`stop_time` DateTime
)
ENGINE = MergeTree()
ORDER BY (start_time, id)
TTL stop_time + toIntervalMonth(1)
Ok.
0 rows in set. Elapsed: 0.014 sec.
db43af298bb9 :) CREATE TABLE chat_messages (id Int64, text String, chat_id Int64) ENGINE = MergeTree() ORDER BY id TTL (SELECT stop_time from chats where chats.id = chat_id) + INTERVAL 1 MONTH;
CREATE TABLE chat_messages
(
`id` Int64,
`text` String,
`chat_id` Int64
)
ENGINE = MergeTree()
ORDER BY id
TTL
(
SELECT stop_time
FROM chats
WHERE chats.id = chat_id
) + toIntervalMonth(1)
Received exception from server (version 19.16.10):
Code: 47. DB::Exception: Received from localhost:9000. DB::Exception: Missing columns: 'chat_id' while processing query: 'SELECT stop_time FROM chats WHERE id = chat_id', required columns: 'id' 'chat_id' 'stop_time', source columns: 'stop_time' 'id' 'start_time'.
0 rows in set. Elapsed: 0.017 sec.
The TTL definition for the second table fails because it tries to find the 'call_id' column in 'chats' table instead of the source 'chat_messages' table. Is what I'm trying to achieve even possible or am I forced to use ALTER DELETE mechanism instead?
Considering a base table employee and a table derived from employee called employee_salary_period which contains a complex datatype map. How to select and insert data from employee into employee_salary_period where salary_period_map is a key value pair i.e. salary: period
CREATE TABLE employee(
emp_id bigint,
name string,
address string,
salary double,
period string,
position string
)
PARTITIONED BY (
dept_id bigint)
STORED AS PARQUET
CREATE TABLE employee_salary_period(
emp_id
name string,
salary string,
period string,
salary_period_map Map<String,String>,
)
PARTITIONED BY (
dept_id bigint)
STORED AS PARQUET
I'm stuck trying to figure out how to select data as salary_period_map
Consider using str_to_map function provided by hive. I hope you have only one key (salary) in you map
select
emp_id
name,
salary,
period,
str_to_map(concat(salary,":",period),'&',':') as salary_period_map
from employee_salary_period