does anyone know how to use a default value like current_timestamp in mysql when creating a clickhouse table? The now() udf is dynamic, instead of the time the row inserted, it is always the current time, it changes when select.
Here is my table:
CREATE TABLE default.test2 (
`num` UInt32,
`dt` String,
`__inserted_time` DateTime DEFAULT now()
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/test2', '{replica}')
PARTITION BY dt
ORDER BY dt
SETTINGS index_granularity = 8192
I want the __inserted_time column value generated automatically thus I don't have to specify it in the insert into test2 (num,dt) values (1,'20191010')
my mistake, DEFAULT now() actually works
Related
I have a table with a column, can i create a partition based on an expression using that column
I read that IBM's Big SQL technology has this feature.
I also know we can partition in hive by a column but what about an expression?
In this case i am doing a cast..it could be any expression
CREATE TABLE INVENTORY_A (
trans_id int,
product varchar(50),
trans_ts timestamp
)
PARTITIONED BY (
cast(trans_ts as date) AS date_part
)
I expect the records to be partitioned by the date value. So I expect that when a user writes a query like
select * from INVENTORY_A where trans_ts BETWEEN timestamp '2016-06-23 14:00:00.000' AND timestamp '2016-06-23 14:59:59.000'
the query will be smart enough to break the timestamp down by the date and do a filter only on the date
You can use Dynamic partitioning and cast your variables in select query.
In my Oracle SQL Developer, i have a table with a column with DATE format. When i insert a new row into this table, and insert a new value in this column, it automatically suggestes me the current date with the current hour.
I would like that it automatically suggestes me current date, but with 00:00:00 hour . Is there some setting or parameter that i can set in my SQL Developer to have this result?
We can't able to insert 00:00:00 hours ... the hour value should be between 1 to 12...
we can use below query to insert 00:00:00 hours but the value will be changed to 12:00:00
INSERT INTO TABLE (DATE_COL) VALUES
( TO_DATE ('11/16/2017 00:00:00 ', 'MM/DD/YYYY HH24:MI:SS '));
It seems to me that your DATE column is set with a DEFAULT of SYSDATE. This means, for any INSERT operations which do not specify a value in your DATE column, the current date and time will populate for that row. However, if INSERT operations do specify a value in your DATE column, then the specified date value will supersede the DEFAULT of SYSDATE.
If an application is controlling INSERT operations on that table, then one solution is to ensure the application utilizes the TRUNC() function to obtain your desired results. For example:
INSERT INTO tbl_target
(
col_date,
col_value
)
VALUES
(
TRUNC(SYSDATE, 'DDD'),
5000
)
;
However, if there are multiple applications or interfaces where users could be inserting new rows into the table, (e.g. using Microsoft Access or users running INSERT statements via SQL Developer) and you can't force all of those interfaces to utilize the TRUNC() function on that column during insertion, then you need to look into other options.
If you can ensure via applications that INSERT operations will not actually reference the DATE, then you can simply ALTER the table so that the DATE column will have a DEFAULT of TRUNC(SYSDATE). A CHECK CONSTRAINT can be added for further integrity:
ALTER TABLE tbl_target
MODIFY
(
col_date DATE DEFAULT TRUNC(SYSDATE, 'DDD') NOT NULL
)
ADD
(
CONSTRAINT tbl_target_CHK_dt CHECK(col_date = TRUNC(col_date, 'DDD'))
)
;
However, if users still have the freedom to specify the DATE when inserting new rows, you will want to use a TRIGGER:
CREATE OR REPLACE TRIGGER tbl_target_biu_row
BEFORE INSERT OR UPDATE OF col_val
ON tbl_target
FOR EACH ROW
BEGIN
:NEW.col_date := TRUNC(SYSDATE, 'DDD');
END tbl_target_biu_row
;
This will take of needing to manage the application code of all external INSERT operations on the table. Keep in mind, the above trigger is also modifying the DATE column if a user updates the specified value column.
Does CockroachDB support default values for columns in its tables? Does it allow default values to be function values (e.g. current_date())?
You can set DEFAULT values using the DEFAULT constraint, which CockroachDB has documented here.
It also supports setting the default value as a function, e.g. to insert the date that a write occurred.
You would create a table with such a default column like:
CREATE TABLE purchase_log (
id INT PRIMARY KEY,
date_purchased DATE DEFAULT current_date()
);
Then all inserts to the table that don't specify the date_purchased column will have the column automatically populated with the return value of current_date() at the time of the insert.
I need help on the following scenario:
1) Memo table is the source table in hive.
It has 5493656359 records.Its desc is as follows:
load_ts timestamp
memo_ban bigint
memo_id bigint
sys_creation_date timestamp
sys_update_date timestamp
operator_id bigint
application_id varchar(6)
dl_service_code varchar(5)
dl_update_stamp bigint
memo_date timestamp
memo_type varchar(4)
memo_subscriber varchar(20)
memo_system_txt varchar(180)
memo_manual_txt varchar(2000)
memo_source varchar(1)
data_dt string
market_cd string
Partition information:
data_dt string
market_cd string
2)
This is the target table
CREATE TABLE IF NOT EXISTS memo_temprushi (
load_ts TIMESTAMP,
ban BIGINT,
attuid BIGINT,
application VARCHAR(6),
system_note INT,
user_note INT,
note_time INT,
date TIMESTAMP)
PARTITIONED BY (data_dt STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS ORC
TBLPROPERTIES ("orc.compress"="SNAPPY");
3)
This is the initial load statement from source table Memo into
target table memo_temprushi. Loads all records till date 2015-12-14:
SET hive.exec.compress.output=true;
SET mapred.output.compression.type=BLOCK;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
INSERT INTO TABLE memo_temprushi PARTITION (DATA_DT)
SELECT LOAD_TS,MEMO_BAN, OPERATOR_ID, APPLICATION_ID,
CASE WHEN LENGTH(MEMO_SYSTEM_TXT)=0 THEN 0 ELSE 1 END,
CASE WHEN LENGTH(MEMO_MANUAL_TXT)=0 THEN 0 ELSE 1 END,
HOUR(MEMO_DATE), MEMO_DATE, DATA_DT
FROM tlgmob_gold.MEMO WHERE LOAD_TS < DATE('2015-12-15');
4)
For incremental load I want to insert the rest of the records i.e. from date 2015-12-15 onward. I'm using following query:
INSERT INTO TABLE memo_temprushi PARTITION (DATA_DT)
SELECT MS.LOAD_TS,MS.MEMO_BAN, MS.OPERATOR_ID, MS.APPLICATION_ID,
CASE WHEN LENGTH(MS.MEMO_SYSTEM_TXT)=0 THEN 0 ELSE 1 END,
CASE WHEN LENGTH(MS.MEMO_MANUAL_TXT)=0 THEN 0 ELSE 1 END,
HOUR(MS.MEMO_DATE), MS.MEMO_DATE, MS.DATA_DT
FROM tlgmob_gold.MEMO MS JOIN (select max(load_ts) max_load_ts from memo_temprushi) mt
ON 1=1
WHERE
ms.load_ts > mt.max_load_ts;
It launches 2 jobs. Initially it gives warning regarding stage being a cross product.
The first job gets completely quite soon but second job remains stuck at reduce 33%.
The log shows : [EventFetcher for fetching Map Completion Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher: EventFetcher is interrupted.. Returning
It shows that the number of reducers is 1.
Trying to increase the number of reducers through this command set mapreduce.job.reduces but it's not working.
Thanks
You can try this.
Run "select max(load_ts) max_load_ts from memo_temprushi"
Add the value in the where condition of the query and remove the join condition of the query.
If it works, then you can develop shell script in which first query will get max value and then run the second query with out join.
Here is the sample shell script.
max_date=$(hive -e "select max(order_date) from orders" 2>/dev/null)
hive -e "select order_date from orders where order_date >= date_sub('$max_date', 7);"
Hey I have two columns in my HIVE table :
For example :-
c1 : name
c2 : age
Now while creating a table I want to declare two more columns which automatically give me the current date and time when the row is loaded.
eg: John 24 26/08/2015 11:15
How can this be done?
Hive currently does not support the feature to add a default value to any column definition while creating a table. Please refer to the link for complete hive create table syntax:
Hive Create Table specification
Alternative work around for this issue would be to temporarily load data into temporary table and use the insert overwrite table statement to add the current date and time into the main table.
Below example may help:
1. Create a temporary table
create table EmpInfoTmp(name string, age int);
2. Insert data using a file or existing table into the EmpInfoTmp table:
name|age Alan|28 Sue|32 Martha|26
3. Create a table which will contain your final data:
create table EmpInfo(name string, age tinyint, createDate string, createTime string);
4. Insert data from the temporary table and with that also add the columns with default value as current date and time:
insert overwrite table empinfo select name, age, FROM_UNIXTIME( UNIX_TIMESTAMP(), 'dd/MM/YYYY' ), FROM_UNIXTIME( UNIX_TIMESTAMP(), 'HH:mm' ) from empinfofromfile;
5. End result would be like this:
name|age|createdate|createtime Alan|28|26/08/2015|03:56 Martha|26|26/08/2015|03:56 Sue|32|26/08/2015|03:56
Please note that the creation date and time values will be entered accurately by adding the data to your final table as and when it comes into the temp table.
Note: You can't set more then 1 column as CURRENT_TIMESTAMP.
Here this way, You cant set CURRENT_TIMESTAMP in one column
SQL:
CREATE TABLE IF NOT EXISTS `hive` (
`id` int(11) NOT NULL,
`name` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`age` int(11) DEFAULT '0',
`datecreated` timestamp NULL DEFAULT CURRENT_TIMESTAMP
);
Hey i found a way to do it using shell script.
Heres how :
echo "$(date +"%Y-%m-%d-%T") $(wc -l /home/hive/landing/$line ) $dir " >> /home/hive/recon/fileinfo.txt
Here i get the date without spaces. In the end I upload the textfile to my hive table.