I seem to be getting the below exception when I try to create a table using Hive client.
create table if not exists test (id int, name string) comment 'test table';
11:15:32.016 [HiveServer2-Background-Pool: Thread-34] ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler - Retrying HMSHandler after 2000 ms (attempt 1 of 10) with error: javax.jdo.JDODataStoreException: Insert of object "org.apache.hadoop.hive.metastore.model.MTable#784fafc2" using statement "INSERT INTO TBLS (TBL_ID,CREATE_TIME,DB_ID,LAST_ACCESS_TIME,OWNER,RETENTION,SD_ID,TBL_NAME,TBL_TYPE,VIEW_EXPANDED_TEXT,VIEW_ORIGINAL_TEXT) VALUES (?,?,?,?,?,?,?,?,?,?,?)" failed : Column 'IS_REWRITE_ENABLED' cannot accept a NULL value.
at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:720)
Caused by: ERROR 23502: Column 'IS_REWRITE_ENABLED' cannot accept a NULL value.
at org.apache.derby.client.am.ClientStatement.completeExecute(Unknown Source)
at org.apache.derby.client.net.NetStatementReply.parseEXCSQLSTTreply(Unknown Source)
I did search but couldn't find a satisfactory resolution.
Here is my set up:
Hive 2.1.0
OS: Windows
Hadoop: 2.9.2
Derby: 10.14.2.0
What am i missing?
Thanks.
Seems like a compatibility issue with derby. I moved back to a earlier version of derby 10.2.1.1 and the issue went away.
Related
Environment:
Apache Hive (version 1.1.0-cdh5.14.2)
I tried creating a table with below DDL.
create external table test1 (v_src_code string,d_extraction_date date) partitioned by (d_mis_date date) row format serde 'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' with serdeproperties ("field.delim"="~|") stored as textfile location '/hdfs_path/test1' tblproperties("serialization.null.format"="");
Then I alter this table by adding one extra column as below.
alter table test1 add columns(n_limit_id bigint);
This is working perfectly fine.
But recently our cluster got upgraded. The new environment is
Apache Hive (version 2.1.1-cdh6.3.4)
The same table is created in this new environment. When I do alter table I get below error.
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Error: type expected at the position 0 of '<derived from deserializer>:bigint' but '<' is found. (state=08S01,code=1)
I want to create hive table on top of phoenix table in emr.
I am facing a NoClassDefFoundError: org.apache.hadoop.hbase.security.SecurityInfo
What I have done so far:
I followed the instructions from https://phoenix.apache.org/hive_storage_handler.html and added phoenix-hive-5.0.0-HBase-2.0.jar to hive-env.sh as well as in hive-site.xml .
Restarted the hive service systemctl restart hive-server2.service
Restarted the metastore systemctl restart hive-hcatalog-server.service
Executed create table command from hue:
create external table ext_table (
i1 int,
s1 string,
f1 float,
d1 decimal
)
STORED BY 'org.apache.phoenix.hive.PhoenixStorageHandler'
TBLPROPERTIES (
"phoenix.table.name" = "ext_table",
"phoenix.zookeeper.quorum" = "localhost",
"phoenix.zookeeper.znode.parent" = "/hbase",
"phoenix.zookeeper.client.port" = "2181",
"phoenix.rowkeys" = "i1",
"phoenix.column.mapping" = "i1:i1, s1:s1, f1:f1, d1:d1"
);
Got an exception: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.security.SecurityInfo)
I am using emr-6.1.0
HBase 2.2.5
Phoenix 5.0.0
Hive 3.1.2
Anybody has an idea what can be the issue?
Update
I followed the advice from #leftjoin and used ADD JAR from hue to add phoenix-hive jar to classpath. Then I faced jar compatibility issue caused by phoenix hive connector that I use:
phoenix-hive-5.0.0-HBase-2.0.jar.
The newer versions of phoenix connectors are not archived into single bundle that could be downloaded from phoenix website . Instead
the connectors are located now in github repo.
I built the new phoenix-hive connector (versions: Phoenix->5.1.0, Hive->3.1.2, Hbase->2.2) and used it to create the Hive table.
As a result I got another exception, which I am not able to fix:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org/apache/phoenix/compat/hbase/CompatSteppingSplitPolicy
I think it is still somehow connected to dependency issues. But no clue what is exactly.
As a workaround put jar into hdfs and execute ADD JAR command before create table and query:
ADD JAR hdfs://path/to/your/jar/phoenix-hive-5.0.0-HBase-2.0.jar;
I am in the process of upgrading from gradle 4 to 6.
This forced me to upgrade many stuff like spring, h2, mySql connector, etc..
Now I have this problem:
on runtime, using MySQL all works fine.
but on tests, I get:
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flyway' defined in com.bluerbn.wallet.infra.SpringTestConfiguration: Invocation of init method failed; nested exception is org.flywaydb.core.internal.command.DbMigrate$FlywayMigrateException:
Migration V9__fixIndex.sql failed
------------------------------------------------
SQL State : 42S22
Error Code : 42122
Message : Column "INDEX" not found; SQL statement:
ALTER TABLE table1 DROP INDEX ACC_INDEX [42122-200]
Location : db/migration/V9__fixIndex.sql (/Users/.../resources/db/migration/V9__fixIndex.sql)
Line : 1
Statement : ALTER TABLE table1 DROP INDEX ACC_INDEX
This is an old script that was working fine before the upgrade.
versions:
spring boot: 2.3.0
org.flywaydb:flyway-core:6.4.1 (though in gradle I put 6.4.2)
com.h2database:h2:1.4.200
mysql:mysql-connector-java:8.0.18
Any ideas?
H2 since the version 1.4.200 allows MySQL-style ALTER TABLE tableName DROP INDEX indexName only in MySQL compatibility mode. In 1.4.199 and older versions this invalid (for H2) command was incorrectly accepted in all modes.
You need to append ;MODE=MySQL to JDBC URL for H2.
I am running a small cluster in Amazone EMR in order to play with Apache Hive 2.3.5. It is my understanding that Apache Hive can import data from a remote database and have the cluster to run queries. I was following an example that is provided in Apache Hive web documentation (https://cwiki.apache.org/confluence/display/Hive/JdbcStorageHandler) and created the following code:
CREATE EXTERNAL TABLE hive_table
(
col1 int,
col2 string,
col3 date
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
'hive.sql.database.type'='POSTGRES',
'hive.sql.jdbc.driver'='org.postgresql.Driver',
'hive.sql.jdbc.url'='jdbc:postgresql://<url>/<dbname>',
'hive.sql.dbcp.username'='<username>',
'hive.sql.dbcp.password'='<password>',
'hive.sql.table'='<dbtable>',
'hive.sql.dbcp.maxActive'='1'
);
But I get the following error:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException java.lang.IllegalArgumentException: Property hive.sql.query is required.)
According to the documentation, I need to specify either “hive.sql.table” or “hive.sql.query” to tell how to get data from jdbc database. But if I replace hive.sql.table with hive.sql.query I get the following error:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException java.lang.IllegalArgumentException: No enum constant org.apache.hive.storage.jdbc.conf.DatabaseType.POSTGRES)
I tried looking in the web for a solution and it doesn't look like anyone experience the same issues that I am having. Do I need to modify a config file or am I missing something critical in my code?
I think you are using a version of the jar which doesn't support POSTGRES.
Download the latest jar from this link:
http://repo1.maven.org/maven2/org/apache/hive/hive-jdbc-handler/3.1.2/hive-jdbc-handler-3.1.2.jar
Put this downloaded jar into a hdfs location.
Run hive normally.
Run command: add jar ${HDFS_PATH_TO_DOWNLOADED_JAR}
Run your create table command
I'm getting the following error while executing queries against a database in impala. With other databases its working fine.
Error trace is as follows.
[Simba][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: select * from test_table limit 1, SQL state: {1}, Query: {2}.[]
java.sql.SQLException: [Simba][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: [Simba][JSQLEngine](12010) The table "test_table" could not be found., SQL state: HY000, Query: select count(*) from test_table.
at com.cloudera.impala.hivecommon.dataengine.HiveJDBCDataEngine.prepare(Unknown Source)
at com.cloudera.impala.jdbc.common.SStatement.executeNoParams(Unknown Source)
at com.cloudera.impala.jdbc.common.SStatement.executeQuery(Unknown Source)
Caused by: com.cloudera.impala.support.exceptions.GeneralException: [Simba][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: [Simba][JSQLEngine](12010) The table "test_table" could not be found., SQL state: HY000, Query: select count(*) from test_table.
... 3 more
if I execute show tables its listing the table name.
If I execute it from hue its not displaying anything in the result.
I tried by invalidating the metadata.
I tried by changing to latest driver jdbc41 same problem.
Where might be the problem?
In my case, this error was caused by not having a /user/scott directory on hdfs with write permissions for the Hiveserver (running as cloudera-scm user)(my jdbc connection uses scott as the user id). Once I created the dir and chmod'ed it, I could run all queries. Earlier only select * worked but select count(*) did not.
The problem was in the .avro file format. My teamlead has fixed it, Not sure what he has done he just said it was the problem of file format.