When run the spark example:
spark-hive-tables , I get errors on hadoop UI
User class threw exception: java.lang.RuntimeException:
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
and warning
executor.CoarseGrainedExecutorBackend: An unknown (x.x.x.x:x) driver
disconnected.
but I have start hive metastore on my spark-yarn cluster, what should I do?
It means you haven't started your Metastore service, so start you metastore service where you installed hive or in the remote if you have your metastore in remote.
To start metastore use hive --service metastore
what output you got after starting metastore service
I found out that i am using thrift server.after starting thrift by cmd
/SPARKPATH/sbin/start-thriftserver.sh ,here comes another error "java.lang.ClassNotFoundException: org.datanucleus.api.jdo.JDOPersistenceManagerFactory" , which display errors like my title here. and it can be fixed by add --jars /SPARKPATH/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar,/SPARKPATH/lib_managed/jars/datanucleus-core-3.2.10.jar,/SPARKPATH/lib_managed/jars/datanucleus-rdbms-3.2.9.jar
Related
I have not initialized Metastore for Hive.
How can I do that? I know I have to run schematool with --service metastore, but am getting an error
Exception in thread "main" java.io.FileNotFoundException: File file:/C:/DB/HADOOP/Hive-3.1.2/bin/nitSchema does not exist
I have installed Hive, added it to PATH and am able to open it using the hive command in Terminal.
However, when I attempt to run a basic command such as
SHOW TABLES;
I am presented with the error:
FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
The instructions I am following do not suggest that anything has to be instantiated.
For reference, I am using the book Hadoop: The Definitive Guide (4th Edition) and running it locally on my machine.
When running JPS the following services are running:
2528 DataNode
7232 RunJar
2441 NameNode
7401 Jps
2634 SecondaryNameNode
282
2842 NodeManager
2751 ResourceManager
I fixed by removing the derby database files
rm -rf $HIVE_HOME/bin/metastore_db
and
$HIVE_HOME/bin/schematool -initSchema -dbType derby
I was able to resolve this problem by initializing the schema. I am surprised it is not mentioned anywhere.
To initialize the schema:
Navigate to your Hive installation folder
[install folder]/bin/schematool -initSchema -dbType derby
Next you should receive some messages confirming
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.derby.sql
Initialization script completed
schemaTool completed
Start hive
Run any basic commands to determine Hive is functioning such as SHOW TABLES;
I am creating a jdbc connection to hive using javax.sql.DataSource and passing zookeeper service discovery (obtained from Ambari) string to Hive .
Zookeeper Hive URL : jdbc:hive2://localhost:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=http;httpPath=cliservice
If I make direct jdbc connection with HiveServer host and port then connection work properly but it fails with zookeeper string.
After that I tested zookeeper string with beeline and I worked fine.
Below is exception when connection is made.
Caused by: java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:205)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:163)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:307)
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:200)
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:710)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:644)
at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:466)
at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:143)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:115)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:102)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:126)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:85)
at com.thinkbiganalytics.kerberos.KerberosUtil.getConnectionWithOrWithoutKerberos(KerberosUtil.java:60)
at com.thinkbiganalytics.hive.service.RefreshableDataSource.getConnectionForValidation(RefreshableDataSource.java:113)
at com.thinkbiganalytics.hive.service.RefreshableDataSource.testAndRefreshIfInvalid(RefreshableDataSource.java:133)
at com.thinkbiganalytics.hive.service.RefreshableDataSource.getConnection(RefreshableDataSource.java:145)
at com.thinkbiganalytics.kerberos.KerberosUtil.getConnectionWithOrWithoutKerberos(KerberosUtil.java:60)
at com.thinkbiganalytics.schema.DBSchemaParser.listCatalogs(DBSchemaParser.java:80)
... 118 more
Caused by: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read HiveServer2 uri from ZooKeeper
at org.apache.hive.jdbc.ZooKeeperHiveClientHelper.getNextServerUriFromZooKeeper(ZooKeeperHiveClientHelper.java:86)
at org.apache.hive.jdbc.Utils.updateConnParamsFromZooKeeper(Utils.java:506)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:203)
... 136 more
Caused by: org.apache.hive.jdbc.ZooKeeperHiveClientException: Tried all existing HiveServer2 uris from ZooKeeper.
at org.apache.hive.jdbc.ZooKeeperHiveClientHelper.getNextServerUriFromZooKeeper(ZooKeeperHiveClientHelper.java:73)
... 138 more
Did anyone encounter this ?
After spending my 2 days , i figured out problem . I have hive 0.14 dependency in my code where this problem is occurring. To fix i updated below two hive maven dependencies..
Hive Services - https://mvnrepository.com/artifact/org.apache.hive/hive-service/1.2.1000.2.4.2.10-1
Hive JDBC - https://mvnrepository.com/artifact/org.apache.hive/hive-jdbc/1.2.1000.2.4.2.10-1
In RedHat test server I installed hadoop 2.7 and I ran Hive ,Pig & Spark with out issues .But when tried to access metastore of Hive from Spark I got errors So I thought of putting hive-site.xml(After extracting 'apache-hive-1.2.1-bin.tar.gz' file I just add $HIVE_HOME to bashrc as per tutorial and everything was working other than this integration with Spark) In apache site I found that I need to put hive-site.xml as metastore configuration
I created the file as below
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:derby://localhost:1527/metastore_db;create=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
</configuration>
I put IP as localhost since it is single node machine .After that I am not able to connect to even Hive .It is throwing error
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
....
Caused by: javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the given database. JDBC url = jdbc:derby://localhost:1527/metastore_db;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
java.sql.SQLException: No suitable driver found for jdbc:derby://localhost:1527/metastore_db;create=true
There are lot many error log pointing to the same thing . If I remove hive-site.xml from the conf folder hive is working without issues .Can anyone point me to the right path for default metastore configuration
Thanks
Anoop R
Derby is used as an embedded database. try using
jdbc:derby:metastore_db;create=true
as jdbc-url. see also
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-EmbeddedMetastore
To use the metastore fully functional (and by that to be able to access it from different services), try setting up using mysql as described in the document above.
As you are setting up an embedded metastore database, use the property below as JDBC URL:
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:derby:metastore_db;create=true </value>
<description>JDBC connect string for a JDBC metastore </description>
</property>
I was also facing similar kind of exception while installing hive. The thing which worked for me was to initialize the derby db. I used the following command to solve the problem : command -> Go to $HIVE_HOME/bin and run the command schematool -initSchema -dbType derby .
You can follow the link http://www.edureka.co/blog/apache-hive-installation-on-ubuntu
It will work if you put derbyclient.jar in lib folder of hive
I am trying to configuring presto 0.60 with Hadoop 2 and Hive remote Metastore. When I try to access the tables "presto cli" through the following error.
./presto.jar --server name:9083 --catalog default --debug
Error:
2014-03-18T18:50:23.739+0100 WARN completer-0 com.google.common.cache.LocalCace
Exception thrown during refresh
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
java.net.SocketTimeoutException: Read timeout
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[presto.jar:0.60]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[presto.jar:0.60]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) [presto.jar:0.60]
at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135) ~[presto.jar:0.60]
at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2346) [presto.at com.google.common.cache.LocalCache$Segment$1.run(LocalCache.java:2329) ~[presto.jar:0.60]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [presto.jar:0.60]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) [presto.jar:0.60]
Thanks
Try this command instead :
./presto.jar --server <presto_coordinator>:8080 --catalog default --debug