My Hive (version 0.14.0.2.2.6.4-1) table table_test is in ORC format with some data, I needed to re-structure that table
I have taken a backup table (ORC format) using
create table table_test_bk as select * from table_test;
Drop Original table as:
drop table table_test;
Running the modified DDL for table_test re-creation (with a new column in the middle column_new string)
The old table struncture:
(col1 string,col2 decimal(10,2),col3 timestamp);
The new table struncture: (col1 string,col2 decimal(10,2),column_new string,col3 timestamp);
Restoring the data-set from backup table using
insert into table table_test select col1,col2,null as column_new,col3 from table_test_bk;
When step 4 runs successfully, dropping up the backup table
All these 5 steps ran successfully, but while doing data-sanity I can see the below exception, and I am unable to get any o/p. Where as doing select count(1) from table_test; is giving me proper data-count.
Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveDecimalWritable cannot be cast to org.apache.hadoop.io.Text
Any help is appreciated.
Related
I created a hive table with a storage handler and now I want to add a column to that table but it gives me below error:
[Code: 10134, SQL State: 42000] Error while compiling statement: FAILED:
SemanticException [Error 10134]: ALTER TABLE can only be used for [ADDPROPS,
DROPPROPS] to a non-native table
As per the hive documentation any hive table you create with storage handler is non native table.
Here's a link https://cwiki.apache.org/confluence/display/Hive/StorageHandlers
There is a JIRA case for enhancement is open with Apache for the same.
https://issues.apache.org/jira/browse/HIVE-1240
For ex, I am using Druid Storage Handler in my case.
I created a hive table using:
CREATE TABLE druid_table_1
(`__time` TIMESTAMP, `dimension1` STRING, `metric1` int)
STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler';
and then I am trying to add a column:
ALTER TABLE druid_table_1 ADD COLUMNS (`dimension2` STRING);
With above approach I am getting an error.
Is there any other way to add a column to non native tables in hive without recreating it?
Patch is available in HDP 2.5+ from Hortonworks. Support for ADD columns has been added in ALTER statement.
Column can be added into druid table using ALTER table DDL in hive.
ALTER TABLE ADD COLUMNS (col_name data_type)
There is no need to specify partition spec as these are druid backed hive tables and partition/storage is maintained by druid.
Can any one pls tell me what is the error in the below query.
insert overwrite directory 'user/cloudera/batch' stored as parquet select * from emp;
I am trying to create parquet table. I am facing the below error when using the above command.
cannot recognize input near 'stored' 'as' 'parquet' in select clause
For creating parquet table first create table and store as parquet.
CREATE TABLE emp (x INT, y STRING) STORED AS PARQUET;
now load the data into this table then you can execute your query.
insert overwrite directory '/user/cloudera/batch' select * from emp;
I'm trying to load a file locally into Hive by running this command:
LOAD DATA INPATH '/data/work/hive/staging/ExampleData.csv' INTO TABLE tablename;
which gives me the error:
SemanticException [Error 10062]: Need to specify partition columns
because the destination table is partitioned (state=42000,code=10062)
An answer I found suggests creating an intermediate table then letting dynamic partitioning kick in to load into a partitioned table.
I've created a table that matches the data and truncated it:
create table temptablename as select * from tablename;
truncate table temptablename
Then loaded the data using:
LOAD DATA INPATH '/data/work/hive/staging/ExampleData.csv' INTO TABLE temptablename;
How do I 'kick in' dynamic partitioning?
1.Load data into temptablename(without partition)
create table temptablename(col1,col2..);
LOAD DATA INPATH '/data/work/hive/staging/ExampleData.csv' INTO TABLE
temptablename;
now once you have data in intermediate table ,you can kick in dynamic
partitioning using following command.
2.INSERT into tablename PARTITION(partition_column) select * from
temptablename;
I'm trying to insert data into my partition table using insert statement but its getting error saying unable to Alter partition
The query that I used :
hive> INSERT INTO TABLE sms_partition table PARTITION(userc,coding) SELECT headerguid,messageid,userid,tonumber,fromnumber,smssplits,udh,operator,circle,vf,msgp,smstext,userc,coding from chotadata16june;
I got the error:
ERROR:Failed with exception org.apache.hadoop.hive.ql.metadata.hiveexception: unable to alter partition .For direct metastore db connections ,we dont support retries to the client level.
I have a external partitioned table with almost 500 partitions. I am trying to create another external table with same properties as of the old table. Then i want to copy all the partitions from my old table to the newly created table. below is my create table query. My old table is stored as TEXTFILE and i want to save the new one as ORC file.
'add jar json_jarfile;
CREATE EXTERNAL TABLE new_table_orc (col1,col2,col3...col27)
PARTITIONED BY (year string, month string, day string)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES (....)
STORED AS orc
LOCATION 'path';'
And after creation of this table. i am using the below query to insert the partitions from old table to new one.i only want to copy few columns from original table to new table
'set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;
INSERT OVERWRITE TABLE new_table_orc PARTITION (year,month,day) SELECT col2,col3,col6,year,month,day FROM old_table;
ALTER TABLE new_table_orc RECOVER PARTITIONS;'
i am getting below error.
'FAILED: SemanticException [Error 10044]: Line 2:23 Cannot insert into target table because column number/types are different 'day': Table insclause-0 has 27 columns, but query has 6 columns.'
Any suggestions?
Your query has to match the number and type of columns in your new table. You have created your new table with 27 regular columns and 3 partition columns, but your query only select six columns.
If you really only care about those six columns, then modify the new table to have only those columns. If you do want all columns, then modify your select statement to select all of those columns.
You also will not need the "recover partitions" statement. When you insert into a table with dynamic partitions, it will create those partitions both in the filesystem and in the metastore.