I was looking for way to add column in Hive table via Beeline interface only when its not present.
create table employee(ename string , eid string);
alter table employee add columns (eid string);
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Duplicate column name: eid (state=08S01,code=1)
is there any way to ignore error or not try adding a column if already present ?
Thanks
Shashi
Related
How do I drop a column of a partiotioned table in Hive?
I have an external table with 4 columns, for example:
column_A
column_B
column_C
dt_test - partition
I have to drop the column_C, so, I'm trying the follow command:
ALTER TABLE TABLE REPLACE COLUMNS(column_A string, column_B string, dt_test timestamp);
The follow error occurs:
Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. Partition column name dt_test conflicts with table columns.
Thanks!
You can drop column_c by the follow command:
ALTER TABLE TABLE REPLACE COLUMNS(column_A string, column_B string);
not having the partition column: dt_test timestamp
When trying to create a distributed table with Citus, it gives a PG::UndefinedColumn: ERROR: column "x" does not exist
I have enabled Citus on workers and main DB:
SELECT run_command_on_workers($cmd$
CREATE EXTENSION citus;
$cmd$);
I created a composite primary key:
ActiveRecord::Base.connection.execute("
ALTER TABLE x DROP CONSTRAINT x_pkey CASCADE;
")
ActiveRecord::Base.connection.execute("
ALTER TABLE x ADD PRIMARY KEY (tenant_id, id);
")
When trying to do:
ActiveRecord::Base.connection.execute("
SELECT create_distributed_table(x, tenant_id);
")
It keeps saying:
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column "x" does not exist
LINE 2: SELECT create_distributed_table(x, tenant_id...
^
Caused by PG::UndefinedColumn: ERROR: column "x" does not exist
Is there something I am forgetting?
The table name and column name need to be passed as text values in single quotes: SELECT create_distributed_table('x', 'tenant_id');
I am trying to add partition to a hive table (partitioned by date)
My problem is that the date needs to be fetched from another table.
My query looks like :
ALTER TABLE my_table ADD IF NOT EXISTS PARTITION(server_date = (SELECT max(server_date) FROM processed_table));
When i run the query hive throws the following error:
Error: Error while compiling statement: FAILED: ParseException line 1:84 cannot recognize input near '(' 'SELECT' 'max' in constant (state=42000,code=40000)
Hive does not allow to use functions/UDF's for the partition column.
Approach 1:
To achieve this you can run the first query and store the result in one variable and then execute the query.
server_date=$(hive -e "set hive.cli.print.header=false; select max(server_date) from processed_table;")
hive -hiveconf "server_date"="$server_date" -f your_hive_script.hql
Inside your script you can use the following statement:
ALTER TABLE my_table ADD IF NOT EXISTS PARTITION(server_date =${hiveconf:server_date});
For more information on the hive variable substitution, you can refer link
Approach 2:
In this approach, you will need to create a temporary table if the partition data you are expecting is already not loaded in any other partitioned table.
Considering your data doesn't have the server_date column.
Load the data into temporary table
set hive.exec.dynamic.partition=true;
Execute the below query:
INSERT OVERWRITE TABLE my_table PARTITION (server_date)
SELECT b.column1, b.column2,........,a.server_date as server_date FROM (select max(server_date) as server_date from ) a, my_table b;
I'm using Hive 2.1.1 and I'm attempting to create a table with . in a column name:
CREATE TABLE `test_table`(
`field.with.dots` string
);
When I do so I get:
FAILED: ParseException line 4:0 Failed to recognize predicate ')'. Failed rule: '[., :] can not be used in column name in create table statement.' in column specification
I must be doing something wrong because the hive documentation says:
In Hive release 0.13.0 and later, by default column names can be specified within backticks (`) and contain any Unicode character (HIVE-6013)
. is a unicode character. And idea what I might be doing?
To give you more context this is on an Amazon EMR 5.5.0 cluster. Thanks!
Source code: HiveParser
...
private char [] excludedCharForColumnName = {'.', ':'};
...
private CommonTree throwColumnNameException() throws RecognitionException {
throw new FailedPredicateException(input, Arrays.toString(excludedCharForColumnName) + " can not be used in column name in create table statement.", "");
}
Jira ticket :Disallow create table with dot/colon in column name
Please note the motivation:
Since we don't allow users to query column names with dot in the
middle such as emp.no, don't allow users to create tables with such
columns that cannot be queried
It seems create table was handled, but not CTAS nor ALTER TABLE...
hive> create table t as select 1 as `a.b.c`;
OK
hive> desc t;
OK
col_name data_type comment
a.b.c int
Time taken: 0.441 seconds, Fetched: 1 row(s)
hive> select * from t;
FAILED: RuntimeException java.lang.RuntimeException: cannot find field a from [0:a.b.c]
hive> create table t (i int);
OK
hive> alter table t change column i `a.b.c` int
hive> select * from t;
Error while compiling statement: FAILED: RuntimeException java.lang.RuntimeException: cannot find field a from [0:a.b.c]
P.s.
I have updated the documentation (look for colon)
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL
I am working on an Advantage Database Server 8.1 and I have created a new table. I want to add a unique constraint for the combination of 2 columns.
I tried
ALTER TABLE TableName
ADD CONSTRAINT ConstraintName
UNIQUE (ColumnName1, ColumnName2)
but I get the error
"ERROR IN SCRIPT: poQuery: Error 7200: AQE Error: State = 42000; NativeError = 2115; [Extended Systems][Advantage SQL Engine]Expected lexical element not found: You are missing the column names. -- Location of error in the SQL
statement is: 33 (line: 2 column: 5)"
Ok the solution I found is:
CREATE UNIQUE INDEX ConstraintName ON TableName (ColumnName1, ColumnName2);