I created a UDF and exported the jar as abc.jar.
Copied the jar in hdfs at /user/hive/warehouse.
Now, I am getting below errors:
hive> ADD JAR /user/hive/warehouse/abc.jar;
/user/hive/warehouse/abc.jar does not exist
Query returned non-zero code: 1, cause: /user/hive/warehouse/abc.jar does not exist.
hive>
When I do, hadoop fs -ls /user/hive, I can see abc.jar at /user/hive/warehouse path.
Where am I doing wrong and what is the solution for this?
When you add jar from hdfs you use the following statement :
ADD jar hdfs://namenode/user/hive/warehouse/abc.jar;
you are not notifying that you are adding the jar from hdfs . That is the cause of your error.
Hope that helps
The way, you are mentioning the path, it will look the file in local file system.
Either place it there, or use hdfs:// like this
hive> ADD JAR /user/hive/warehouse/abc.jar => local filesystem
hive> ADD JAR hdfs://namenodei/user/hive/warehouse/abc.jar => In hdfs
Above options are valid for current sessions only. So every time you need to write ADD JAR.
In order to add it permanently recommended ways are as follows.
add in hive-site.xml
<property>
<name>hive.aux.jars.path</name>
<value>file://localpath/yourjar.jar</value>
</property>
Copy and paste the JAR file to the ${HIVE_HOME}/auxlib/ folder
Related
I have created a hive UDF JAR file and I am trying to deploy it. For this, I have put all the files into my edge node location /opt/hive/jars and set this path in hive-site.xml file.
<property>
<name>hive.aux.jars.path</name>
<value>/opt/hive/jars</value>
</property>
I have restarted by hive server to using following command
sudo restart hive-server2
However when i login to my beeline I am not able to see jars. When I create a function and call it it's giving an error.
Update 1:
I put the file on hdfs and included that location as well. No luck.
I included the same property in /etc/hive/conf/hiveserver2-site.xml but no luck.
Directory where Jars are located in owned by hive user and has 777 permission.
Update 2:
I checked from which path the other jars files are being picked up.
I put my jars files into these location and restarted the hive server. And Now it's working.
I'm trying to add hive-contrib-0.10.0.jar in hive using ADD JAR hive-contrib-0.10.0.jar command but it always saying hive-contrib-0.10.0.jar does not exist.
I'm using HDP 2.1 version right now. I also added this jar file into /user/root folder using hue and run the command
ADD JAR hdfs:///hive-contrib-0.10.0.jar
but it giving me same error jar file doesn't exist.
Is there any way to solve this problem.
Where should I keep this jar file so that it will run successfully and what will be the command to be used?
upload the JAR file into hdfs path
Add the JAR File using Add command and HDFS full PATH
Example:
hadoop fs -put ~/Downloads/hive.jar /lib/
open hive shell
add jar hdfs:///lib/hive.jar
I see following issues with your approach. Before adding make sure you are able to list the file on Local file system or hdfs where ever it exists.
The jar you are trying to add is by default in hive class path as its part of $HIVE_HOME/lib (on local file system where ever you have hive client/service installed)
on the other hand in regards to your question about how to add jars in hive, we can add using local file system or hadoop distributed file system (HDFS)
Add jar file:///root/hive-contrib-0.10.0.jar (Given that you copied this jar on LFS root directory)
Add jar hdfs://<namenode_hostname>:8020/user/root/hive-contrib-0.10.0.jar (Given that you copied to HDFS root home)
if you want to permanently add the jars you need to do the following.
1. Hive-site.xml ( /etc/hive/conf )
<property>
<name>hive.aux.jars.path</name>
<value>file:///mnt1/hive-jars/hive-contrib-2.1.1.jar</value>
</property>
add hive-contrib-2.1.1.jar to the path "/mnt1/hive-jars" configured in hive-site.xml
This should ideally work after restarting hive-server2.
3. sudo stop hive-server2
4. sudo start hive-server2
But sometimes it does not work. i am not sure why so you can use the following dirty way.
put your jar file in the following path so that hive automatically picks it up while restart.
add hive-contrib-2.1.1.jar to /usr/lib/hive-hcatalog/share/hcatalog
sudo stop hive-server2
sudo start hive-server2
I have read these answers above which was very useful. And I combined all into one solution:
put jars into local disk and give read/write permission
chmod -R 777 /tmp/json.jar
upload to hdfs file system and give permissions too:
hdfs dfs -put /tmp/json.jar hdfs://1.1.1.1:8020/jars/
hdfs dfs -chmod -R 777 hdfs://1.1.1.1:8020/jars/
add jar into hive env.
add jar hdfs://1.1.1.1:8020/jars/json.jar
You have to give the full path to the jar JAR and not only its name.
Don't guess the location. Check the file system to see that it is there, before trying to add it.
I'm running an SQL query on a JSON serde table. It's working in the Hive CLI, but it's failing in Hue with the error:
Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
I guess it's due to the missing jar file; any idea how to add the jar file hive-hcatalog-core-1.2.1.jar for Hue?
Place your jar in HDFS and add same path by using ADD JAR hdfs:///user/hive/lib/hive-hcatalog-core-1.2.1.jar ;
Run ADD JAR hive-hcatalog-core-1.2.1.jar in hue before your query this thing will be present till your current secession persists.
For the benefit of others, who might face same issue either for this particular jar "hive-hcatalog-core-1.2.1.jar" or any udf jar:
In the HUE - Query Editor, run the following command:
add jar hdfs:/hive-hcatalog-core-1.2.1.jar;
Please note single quotes is not required as is the case with Hive CLI
Exact command cloudera gave is ADD JAR {{lib_dir}}/hive/lib/hive-contrib.jar;
1)I am unable to find hive/lib directory on CDH 5
The {{lib_dir}} on CDH installed environments for Hive would either be /usr/lib/hive/ or /opt/cloudera/parcels/CDH/lib/hive/ (depending on packages or parcels being in use).
this is the way to add jar in cloudera
for this you have to change to supper user by use this command
SUDO SU
it will change to supper user
I have to add a UDF in hive.
The query I am trying is :
create function strip1 as 'com.hadoopbook.hive.Strip' using jar '/home/hduser/Hadoop-tutorial/hadoop-book-master/ch17-hive/src/main/java/com/hadoopbook/hive/Strip.jar'
But I am getting a exception as :
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask. Hive warehouse is non-local, but /home/hduser/Hadoop-tutorial/hadoop-book-master/ch17-hive/src/main/java/com/hadoopbook/hive/Strip.jar specifies file on local filesystem. Resources on non-local warehouse should specify a non-local scheme/path
Can anyone tell how to solve this ?
Three options:
copy the jar on hdfs and use that path.
OR
as error is telling you: In the $HIVE_HOME/conf directory there is the hive-default.xml and/or hive-site.xml which has the hive.metastore.warehouse.dir property. add hdfs:/ to this path, and restart/re-run the hive shell/script:
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://usr/hive/warehouse </value>
<description>location of the warehouse directory</description>
</property>
OR
if you are running hive queries from hive shell then:
hive> set hive.metastore.warehouse.dir;
hive.metastore.warehouse.dir=/user/hive/warehouse
above command prints the path, just prefix the hdfs:/ to it as below and then re-run your hive command(s) :
hive> set hive.metastore.warehouse.dir="hdfs://user/hive/warehouse";
You could setting the configuration hive.aux.jars.path to /home/hduser/Hadoop-tutorial/hadoop-book-master/ch17-hive/src/main/java/com/hadoopbook/hive/
and create hive udf function via below command:
create function strip1 as 'com.hadoopbook.hive.Strip'
You can first try to add UDF jar to a hdfs location instead of the local directory:
$ add jar "hdfs://user/cloudera/hive/udf/Strip.jar"
and then create hive function as below:
$ create function test_function as "com.hadoopbook.hive.Strip"
Hope this helps :)
Is there any way I can add hive jars permanently instead of adding at session level in hive shell?
Any help would be appreciated
In the hiveserver2 host, create a location something like /var/lib/hive and add all the necessary jars inside that folder. Edit the hive-site.xml and mention all these jars in the property hive.aux.jars.path
Eg:
ADD JAR /home/amal/hive/amaludf.jar
ADD JAR /home/amal/hive/amaludf2.jar
Instead of using the above commands in each session, you can define it for all sessions.
Create a location for storing these jars in the hiveserver host.
mkdir /var/lib/hive
Add all these jars to that directory
Set the property in hive-site.xml
<property>
<name>hive.aux.jars.path</name>
<value>/var/lib/hive</value>
</property>
Restart the hiveserver2 after doing this modification.
Instead of creating a directory and putting all the jars, you can specify paths of individual jars also. The only condition is that all these jars should be present in the hiveserver host.
Eg:
<property>
<name>hive.aux.jars.path </name>
<value>file:///home/amal/hive/udf1.jar,file:///usr/lib/hive/lib/hive-hbase-handler.jar</value>
</property>
You will have to put the jar in the lib folder of hadoop or hive in all your nodes.
these can be done by two steps
Hive Client should be avalable in all nodes.
Hive Live location should be defined in hadoop-env.sh CLASSPATH and the same file should be updated in entired Hadoop Clueter.
{hadoop-env.sh should be update with CLASSPATH of hive and other location for user defined custom jars and common location which available in entire cluster }
You also need to restart the hive/hadoop to take effect if after changes it dnt work.
create directory named auxlib in $HIVE_HOME, put all your jars in this directory and restart the hive server. run ps -ef | grep hive this command to list hive processes, search for hive.aux.jars.path and you will see that all your jars will be listed against this hiveconf.