What should I specify in config.xml to enable (set to 1) the allow_drop_detached parameter in the system.settings table?
It cannot be changed by 'alter table' request.
Error message:
Code: 344. DB::Exception: Received from localhost:9000. DB::Exception: Cannot execute query: DROP DETACHED PART is disabled (see allow_drop_detached setting).
It should be defined in user.xml not config.xml.
Create an arbitrary file with required changes (here it is query.settings.xml) in directory /etc/clickhouse-server/users.d/:
nano /etc/clickhouse-server/users.d/query.settings.xml
with content (here this param is set only for default-profile):
<?xml version="1.0"?>
<yandex>
<profiles>
<default>
<!-- Allow ALTER TABLE ... DROP DETACHED PART[ITION] ... queries. -->
<allow_drop_detached>1</allow_drop_detached>
</default>
</profiles>
</yandex>
Save this file and check that param is set to 1:
SELECT *
FROM system.settings
WHERE name = 'allow_drop_detached'
/*
┌─name────────────────┬─value─┬─changed─┬─description─────────────────────────────────────────────────┬─min──┬─max──┬─readonly─┐
│ allow_drop_detached │ 1 │ 1 │ Allow ALTER TABLE ... DROP DETACHED PART[ITION] ... queries │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │ 0 │
└─────────────────────┴───────┴─────────┴─────────────────────────────────────────────────────────────┴──────┴──────┴──────────┘
*/
The alternate ways to set this param are:
query level
ALTER TABLE table
DROP DETACHED PARTITION '2020-04-01'
SETTINGS allow_drop_detached = 1;
session level
SET allow_drop_detached = 1;
ALTER TABLE table
DROP DETACHED PARTITION '2020-04-01';
Related
Is it possible to alter a table engine in clickhouse table like in MySQL, something like this:
CREATE TABLE example_table (id UInt32, data String) ENGINE=MergeTree() ORDER BY id;
ALTER example_table ENGINE=SummingMergeTree();
Because I didn't find such capability in the documentation.
If it is not possible, are there any plans to implement it in near future, or what architecture limitations prevent from doing this?
It's possible to change an Engine by several ways.
But it's impossible to change PARTITION BY / ORDER BY. That's why it's not documented explicitly. So in 99.99999% cases it does not make any sense. SummingMergeTree uses table's ORDER BY as a collapsing rule and the existing ORDER BY usually does not suit.
Here is an example of one the ways (less hacky one),
(you can copy partitions from one table to another, it's almost free operation, it exploits FS hardlinks and does not copy real data). (COW -- copy on write).
CREATE TABLE example_table (id UInt32, data Float64)
ENGINE=MergeTree() ORDER BY id;
Insert into example_table values(1,1), (1,1), (2,1);
CREATE TABLE example_table1 (id UInt32, data Float64)
ENGINE=SummingMergeTree() ORDER BY id;
-- this does not copy any data (instant & almost free command)
alter table example_table1 attach partition tuple() from example_table;
SELECT * FROM example_table1;
┌─id─┬─data─┐
│ 1 │ 1 │
│ 1 │ 1 │
│ 2 │ 1 │
└────┴──────┘
optimize table example_table1 final;
select * from example_table1;
┌─id─┬─data─┐
│ 1 │ 2 │
│ 2 │ 1 │
└────┴──────┘
One more way (edit metadata file, also ZK records if a table Replicated)
detach table example_table;
vi /var/lib/clickhouse/metadata/default/example_table.sql
replace MergeTree with SummingMergeTree
attach table example_table;
SHOW CREATE TABLE example_table
┌─statement──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ CREATE TABLE default.example_table
(
`id` UInt32,
`data` Float64
)
ENGINE = SummingMergeTree
ORDER BY id
SETTINGS index_granularity = 8192 │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
SELECT * FROM example_table;
┌─id─┬─data─┐
│ 1 │ 1 │
│ 1 │ 1 │
│ 2 │ 1 │
└────┴──────┘
optimize table example_table final;
SELECT * FROM example_table;
┌─id─┬─data─┐
│ 1 │ 2 │
│ 2 │ 1 │
└────┴──────┘
I have several Tables and materialized views which haven't been created with To [db] statement and have an inner tables with these names:
│ .inner_id.27e007b7-d831-4bea-8610-b3f0fd1ee018 │
│ .inner_id.3e27a1eb-f40f-4c6d-bc70-7722f2cc953b │
│ .inner_id.53c0fe00-151a-4bb5-ba61-237da8c1c6f2 │
│ .inner_id.7785fe45-94e0-42ee-8c0e-14f2d74e3c88 │
│ .inner_id.7d7e6485-e18e-47e0-9bb6-51a31a8cafd0 │
│ .inner_id.cd920d50-7469-4507-a5cf-244de054b037 │
│ .inner_id.fe6d3ce5-7ffc-4bca-9d6b-4015b1807e4f │
Which is different from the name of materialized views i used to create them.
Is there any way to get inner table name of materialized views with clickhouse query?
To resolve uuid name use this query:
SELECT
uuid,
name
FROM system.tables
WHERE database = '{name_of_database}' AND engine = 'MaterializedView'
/*
┌─────────────────────────────────uuid─┬─name───────┐
│ 6b297d75-1c67-4824-9a60-94195c160436 │ mv_name_01 │
..
└──────────────────────────────────────┴────────────┘
*/
Clickhouse offical document implies one user can only have one profile, and database is configured in user section, readonly is configured in profile section, so, readonly will effect on all user's database.
So, how can I set a user x which only can read database dbA, but can read and write database dbB?
Consider using the SQL-driven access control management:
Prepare test environment:
CREATE DATABASE db_01;
CREATE DATABASE db_02;
CREATE TABLE db_01.table_01 (id Int32) Engine = Memory();
CREATE TABLE db_02.table_02 (id Int32) Engine = Memory();
INSERT INTO db_01.table_01 VALUES (1), (2);
INSERT INTO db_02.table_02 VALUES (10), (20);
Create user and assign the required privileges:
CREATE USER user_01 HOST ANY PROFILE 'default';
/* Set readonly access to DB 'db_01'. */
GRANT SELECT ON db_01.* TO user_01;
/* Set read-write access to DB 'db_02'. */
GRANT SELECT ON db_02.* TO user_01;
GRANT INSERT ON db_02.* TO user_01;
Test (need to login as user_01):
SELECT currentUser(); /* Make sure that user is right. */
/*
┌─currentUser()─┐
│ user_01 │
└───────────────┘
*/
SHOW GRANTS; /* Double check the assigned privileges. */
/*
┌─GRANTS─────────────────────────────────────┐
│ GRANT SELECT ON db_01.* TO user_01 │
│ GRANT INSERT, SELECT ON db_02.* TO user_01 │
└────────────────────────────────────────────┘
*/
SELECT * FROM db_01.table_01; /* OK */
INSERT INTO db_01.table_01 VALUES (3); /* DB::Exception: user_01: Not enough privileges. */
SELECT * FROM db_02.table_02; /* OK */
INSERT INTO db_02.table_02 VALUES (30); /* OK */
The code above is just for the demo, you need to consider introduce roles/quotas/.., inherit another user-profile (more restrictive, not 'default') etc.
I have a column in a table with data type Int32. Is it possible to convert column into array data type Array(Int32). If not what are the other ways, kindly let me know.
The changing type of a column from int to Array(int) cannot be performed by ALTER TABLE .. MODIFY COLUMN-query because such typecasting is not allowed.
So you need to follow these steps:
add the new column with type Array(int)
ALTER TABLE test.test_004 ADD COLUMN `value_array` Array(int);
/* Test table preparation:
CREATE TABLE test.test_004
(
`id` int,
`value` int
)
ENGINE = MergeTree();
INSERT INTO test.test_004 VALUES (1, 10), (2, 20), (3, 30), (4, 40);
*/
update array-column value
ALTER TABLE test.test_004
UPDATE value_array = [value] WHERE 1
/* Result
┌─id─┬─value─┬─value_array─┐
│ 1 │ 10 │ [10] │
│ 2 │ 20 │ [20] │
│ 3 │ 30 │ [30] │
│ 4 │ 40 │ [40] │
└────┴───────┴─────────────┘
*/
important: depends on the count of rows in the table this operation can take much time to execute. To check the status of update (mutation) or find the reason of failing observe the system.mutations-table
SELECT *
FROM system.mutations
WHERE table = 'test_004'
when the mutation be completed the original column can be removed
ALTER TABLE test.test_004
DROP COLUMN value
Remark: assign the extra ON CLUSTER-clause for each query if the table is located on several servers.
I have a table in clickhouse, say "my_table", which has replicates (my_table_rep1,...). And I need to add a column, of type float64, with default value (-1).
How should I do that?
I would prefer that the default are not actually added to existing entries.
The documentation is pretty straight-forward:
ALTER TABLE [db].name [ON CLUSTER cluster] ADD COLUMN name [type] [default_expr] [AFTER name_after]
Regarding that:
I would prefer that the default are not actually added to existing entries.
There's also a statement in the docs:
If you add a new column to a table but later change its default expression, the values used for old data will change (for data where values were not stored on the disk)
So, basically, you have to:
Add a column (without a default value, or with DEFAULT 0, or with something else - depends on what do you want to have in the existing entries)
Do OPTIMIZE TABLE .. FINAL to force Clickhouse to write the new data to the disk
Modify the column and set a DEFAULT -1 so only the new rows would be affected
An example:
:) CREATE TABLE my_table (date Date DEFAULT today(), s String) ENGINE = MergeTree(date, (date), 8192);
:) INSERT INTO my_table (s) VALUES ('1. foo');
:) ALTER TABLE my_table ADD COLUMN f Float64;
:) INSERT INTO my_table (s) VALUES ('2. bar');
:) SELECT * FROM my_table;
┌───────date─┬─s──────┬─f─┐
│ 2018-04-20 │ 1. foo │ 0 │
│ 2018-04-20 │ 2. bar │ 0 │
└────────────┴────────┴───┘
:) OPTIMIZE TABLE my_table PARTITION 201804 FINAL;
:) ALTER TABLE my_table MODIFY COLUMN f Float64 DEFAULT -1;
:) INSERT INTO my_table (s) VALUES ('3. baz');
:) SELECT * FROM my_table;
┌───────date─┬─s──────┬──f─┐
│ 2018-04-20 │ 3. baz │ -1 │
│ 2018-04-20 │ 1. foo │ 0 │
│ 2018-04-20 │ 2. bar │ 0 │
└────────────┴────────┴────┘
You really have to do OPTIMIZE TABLE ... FINAL, because if you won't do that, weird things will happen: https://gist.github.com/hatarist/5e7653808e59349c34d4589b2fc69b14