I've removed my HDFS path /user/abc, and some Hive tables were stored in /user/abc/data/abc.db , with a rm -R command.
Despite having my regular tables correctly deleted with Hive SQL, my external tables didn't drop, with the following error:
[Code: 1, SQL State: 08S01] Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Failed to load storage handler: Error in loading storage handler.org.apache.phoenix.hive.PhoenixStorageHandler)
How can I safely delete the tables?
I tried using:
delete from TBL_COL_PRIVS where TBL_ID=[myexternaltableID];
delete from TBL_PRIVS where TBL_ID=[myexternaltableID];
delete from TBLS where TBL_ID=[myexternaltableID];
But it didn't work with the following error message:
[Code: 10297, SQL State: 42000] Error while compiling statement: FAILED: SemanticException [Error 10297]: Attempt to do update or delete on table sys.TBLS that is not transactional
Thank you,
NB: I know a schema is supposed to be deleted more safely with HiveQL but on this particular case, this was not done this way.
Solution is to delete the tables from Hive Metastore (PostgreSQL) with
delete from "TABLE_PARAMS" where "TBL_ID"='[myexternaltableID]';
delete from "TBL_COL_PRIVS" where "TBL_ID"='[myexternaltableID]';
delete from "TBL_PRIVS" where "TBL_ID"='[myexternaltableID]';
delete from "TBLS" where "TBL_ID"='[myexternaltableID]';
NB: Order is important.
Related
The table is of parquet format and sits in Minio storage. I create the parquet file. After the parquet file is inserted into Minio, I run the following command to add the new partition:
call system.sync_partition_metadata('myschema', 'mytable', 'ADD', true)
However, it errors out
[Code: 65551, SQL State: ] Query failed (#20221210_041833_00033_4iqfp): unexpected partition name: mypartitionfield=HIVE_DEFAULT_PARTITION != []
Earlier on I did insert an empty partition value. How can I remedy this issue? The mypartitionfield is of type date.
I tried dropping and creating the table, as well as deleting the Minio folder structure and creating it again. None worked.
I want to create hive database using the command:
create database sbx_products_diff
But it fails with the following error:
jdbc:hive2://myhost> create database sbx_products_diff;
INFO : Compiling
command(queryId=hive_20220414115555_9db94201-d1b7-40ca-8cee-d1d4147c080e):
create database sbx_products_diff INFO : Semantic Analysis
Completed INFO : Returning Hive schema: Schema(fieldSchemas:null,
properties:null) INFO : Completed compiling
command(queryId=hive_20220414115555_9db94201-d1b7-40ca-8cee-d1d4147c080e);
Time taken: 0.003 seconds INFO : Executing
command(queryId=hive_20220414115555_9db94201-d1b7-40ca-8cee-d1d4147c080e):
create database sbx_products_diff INFO : Starting task
[Stage-0:DDL] in serial mode ERROR : FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Database
sbx_products_diff already exists INFO : Completed
executing
command(queryId=hive_20220414115555_9db94201-d1b7-40ca-8cee-d1d4147c080e);
Time taken: 0.014 seconds Error: Error while processing statement:
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. Database
sbx_products_diff already exists (state=42000,code=1)
But the problem is that this database doesn't exists actually:
jdbc:hive2://myhost> drop database sbx_products_diff;
Error: Error while compiling statement:
FAILED: SemanticException [Error 10072]: Database does not exist:
sbx_products_diff (state=42000,code=10072)
Seems like this problem is related to this database particularly, because if I run exactly the same command, but with another name it works perfectly, as expected. Could someone make a direction what is the root of my troubles? Or what should I share to let you help me?
UPD1:
If you look in the hive warehouse folder, do you see a folder called "sbx_products_diff"?
No, the directory doesn't exist. But I should tell another thing, that is interesting. _diff part of the name is suffix. I mean there are several existing databases - sbx_products, sbx_products_aux, sbx_products_bkp, .. And all of those databases located in hdfs with the following structure:
hdfs://bla-bla/sbx_products/hive/pa
hdfs://bla-bla/sbx_products/hive/aux
hdfs://bla-bla/sbx_products/hive/bkp
...
Also, I have already searched my database with commands like:
hdfs dfs -ls -R / | grep diff
hdfs dfs -ls -R / | grep sbx_products
But nothing related to my ghost database were found.
I am using hive version 3.1.0 in my project I have created one external table using below command.
CREATE EXTERNAL TABLE IF NOT EXISTS testing(ID int,DEPT int,NAME string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE;
I am trying to create an index for the same external table using the below command.
CREATE INDEX index_test ON TABLE testing(ID)
AS 'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler'
WITH DEFERRED REBUILD ;
But I am getting below error.
Error: Error while compiling statement: FAILED: ParseException line 1:7 cannot recognize input near 'create' 'index' 'user_id_user' in ddl statement (state=42000,code=40000)
According to Hive documentation, Hive indexing is removed since version 3.0
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Indexing#LanguageManualIndexing-IndexingIsRemovedsince3.0
I am running the insert script in hive by load '.csv' file from local file system into hive table as below:
load data local inpath 'xxx.csv' into table xxx;
I got an error say
Failed with exception Unable to move source
file:/home/hadoop/hbase-data/xxx.csv to destination
hdfs://xxx.xxx.xxx:8020/test/xxx.csv FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
Can anyone help me out with this?
Thanks so much for your effort.
I am getting the below error while copying data into hive table - I have given g+w permissions to user/hive/warehouse and /tmp.
Failed with exception Unable to move source file:/home/vm4learning/Desktop/vehicletesting.txt to destination hdfs://localhost:9000/user/hive/warehouse/imdb.db/vts/vehicletesting.txt
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
but you're trying to copy to
hdfs://localhost:9000/user/hive/warehouse/imdb.db/vts
so you should check that directory's permissions.