I am trying to execute my hive from command prompt.
When i am trying to run the command on my Windows 10 machine.
i.e C:\hadoop-2.7.1\hive-2.1.0\bin>hive
It throws the Error applying authorization policy on hive configuration Error.
Here is full Stack of Error:
Error applying authorization policy on hive configuration: Couldn't create directory ${system:java.io.tmpdir}\${hive.session.id}_resources
What could be the problem.
Check if namenode and datanode services are running. Start those by sbin/start-dfs.cmd on windows.
Related
I'm using Nifi 1.6.0.
I'm trying to write to HDFS and to Hive (cloudera) with nifi.
On "PutHDFS" I'm configure the "Hadoop Confiugration Resources" with hdfs-site.xml, core-site.xml files, set the directories and when I'm trying to Start it I got the following error:
"Failed to properly initialize processor, If still shcedule to run,
NIFI will attempt to initalize and run the Processor again after the
'Administrative Yield Duration' has elapsed. Failure is due to
java.lang.reflect.InvocationTargetException:
java.lang.reflect.InvicationTargetException"
On "PutHiveStreaming" I'm configure the "Hive Metastore URI" with
thrift://..., the database and the table name and on "Hadoop
Confiugration Resources" I'm put the Hive-site.xml location and when
I'm trying to Start it I got the following error:
"Hive streaming connect/write error, flow file will be penalized and routed to retry.
org.apache.nifi.util.hive.HiveWritter$ConnectFailure: Failed connectiong to EndPoint {metaStoreUri='thrift://myserver:9083', database='mydbname', table='mytablename', partitionVals=[]}:".
How can I solve the errors?
Thanks.
For #1, if you got your *-site.xml files from the cluster, it's possible that they are using internal IPs to refer to components like the DataNodes and you won't be able to reach them directly using that. Try setting dfs.client.use.datanode.hostname to true in your hdfs-site.xml on the client.
For #2, I'm not sure PutHiveStreaming will work against Cloudera, IIRC they use Hive 1.1.x and PutHiveStreaming is based on 1.2.x, so there may be some Thrift incompatibilities. If that doesn't seem to be the issue, make sure the client can connect to the metastore port (looks like 9083).
When i am trying to read external table using GPHDFS Protocol. Additionally, I am not able to access HDP2.X files via greenplum cluster.
Getting Error
devdata=# select count(*) from schema.ext_table;
ERROR: external table gphdfs protocol command ended with error. Error occurred during initialization of VM (seg5 slice1 datanode0:40001 pid=13407)
DETAIL:
java.lang.OutOfMemoryError: unable to create new native thread
Command: 'gphdfs://Authorithy/path
More symptoms
Not able to run Hadoop list file command from gpadmin user at greenplum cluster.
that is
gpadmin$hdfs dfs -ls hdfs://namenode/file/path
We tried :
checked Setting related to gphdfs vm paramteres.
I am getting the following error: (through command line as well as web-interface).
Useful info:
1. Hive, HDFS, Yarn services are up and running.
2. I can even get into hive prompt through command line and web-interface. The error occurs when I use show databases. (or click refresh symbol on database explorer of web-interface).
3. I logged in as root user, hdfs user
4. I tried changing permissions to 755 for the directory /user/root
Any help would be greatly appreciated..
------------------start of error message (copied from that of web-interface log)
Unable to submit statement. Error while processing statement: FAILED: Hive Internal Error: com.sun.jersey.api.client.ClientHandlerException(java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 503, message: Service Unavailable) [ERROR_STATUS].
Step 1) Restart Atlas on Sandbox.
Step 2) Restart Hive services on Sandbox.
For me this resolved the issue.
Cheers
I am working on Hive-jdbc connection in HDP 2.1
Code is working fine for queries where mapreduce is not involved like "select * from tabblename". The same code is showing error when the query is modified with a 'where' clause or if we specify columnnames(which will run mapreduce in the the background).
I have verified the correctness of the query by executing it in HiveCLI.
Also I have verified the read/write permissions for the table for the user through which I am running the java-jdbc code.
The error is as follows
java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:275)
at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:355)
at com.testing.poc.hivejava.HiveJDBCTest.main(HiveJDBCTest.java:25)
Today I also got this exception when I submit a hive task from java.
The following error:
org.apache.hive.jdbc.HiveDriverorg.apache.hive.jdbc.HiveDriverhive_driver:
org.apache.hive.jdbc.HiveDriverhive_url:jdbc:hive2://10.174.242.28:10000/defaultget
connection sessucess获取hive连接成功！
java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
I tried to use the sql execute in hive and it works well. Then I saw the log in /var/log/hive/hadoop-cmf-hive-HIVESERVER2-cloud000.log.out then I found the reason of this error. The following error:
Job Submission failed with exception 'org.apache.hadoop.security.AccessControlException(Permission denied: user=anonymous, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
Solution
I used the following command :
sudo -u hdfs hadoop fs -chmod -R 777 /
This solved the error!
hive_driver:org.apache.hive.jdbc.HiveDriver
hive_url:jdbc:hive2://cloud000:10000/default
get connection sessucess
获取hive连接成功！
Heart beat
执行insert成功！
If you use beeline to execute the same queries, do you see the same behaviour as you get while running your test program?
The beeline client also uses the open source JDBC driver and connects to Hive server, which is similar to what you do in your program. HiveCLI on the other hand has Hive embedded in it and does not connect to a remote Hive server by default. You can use HiveCLI to connect to a remote Hive Server 1 but I don't believe you can use it to connect to Hive Server2 (use beeline for Hive Server 2).
For this error, you can take a look at the hive.log and hiveserver2.log on the server side to get more insight into what might have caused the MapReduce error.
Hope this helps.
Cheers,
Holman
I am trying to access Cloudera Hadoop setup (HIVE + Impala) from Mac Book Pro OS X 10.8.4.
We have Cloudera CDH-4.3.0 installed on Linux servers. I have extracted CDH-4.2.0 tarball to my Mac Book Pro.
I have set proper configuration and Kerberos credentials so that commands like 'hadoop -fs -ls /' works and HIVE shell starts up.
However when I do 'show databases' command it gives following error:
> hive
> show databases;
>
Failed with exception java.io.IOException:java.io.IOException: Can't get Master Kerberos principal for use as renewer
The error is related to TokenCache.
When I searched for error, it seems following method 'obtainTokensForNamenodesInternal' throws this error when it tries to get a delegation token for specific FS and fails.
http://hadoop.apache.org/docs/current/api/src-html/org/apache/hadoop/mapreduce/security/TokenCache.html
On client side I don't see any error in HIVE shell logs. I have also tried using tarballs of CDH 4.3.0 with same configuration I get the same error.
Any help or pointers for resolving this error would be highly appreciated.
It seems that you have not config the kerberos for yarn.
Add the follow configure in your yarn-site.cml
<property>
<name>yarn.nodemanager.principal</name>
<value>yarn_priciple/fqdn#_HOST</value>
</property>
<property>
<name>yarn.resourcemanager.principal</name>
<value>yarn_priciple/fqdn#_HOST</value>
</property>
Create a new Gateway YARN role instance in the host from Cloudera Manager. It will automatically setup and update the yarn-site.xml.