i am getting the following error while running the following oozie command
hadoop#master1:~/work/oozie-4.1.0/bin$ oozie-setup.sh -hadoop 0.20.200 $HADOOP_HOME -extjs /home/hadoop/work/ext-2.2.zip
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
Usage : oozie-setup.sh
prepare-war [-d directory] [-secure] (-d identifies an alternative directory
for processing jars
-secure will configure the war file to use HTTPS (SSL))
sharelib create -fs FS_URI [-locallib SHARED_LIBRARY] (create sharelib for
oozie,
FS_URI is the fs.default.name
for hdfs uri; SHARED_LIBRARY, path to the
Oozie sharelib to install, it can be a tarball
or an expanded version of it. If ommited,
the Oozie sharelib tarball from the Oozie
installation directory will be used)
(action failes if sharelib is already installed
in HDFS)
sharelib upgrade -fs FS_URI [-locallib SHARED_LIBRARY] (upgrade existing
sharelib, fails if there
is no existing sharelib installed in HDFS)
db create|upgrade|postupgrade -run [-sqlfile ] (create, upgrade or postupgrade oozie db with an
optional sql File)
(without options prints this usage information)
EXTJS can be downloaded from http://www.extjs.com/learn/Ext_Version_Archives
Any idea to solve this?
EXT.js iw for WEBUI right,
It shows the we need to Prepare war that means need to add the EXT.js by passing the command
oozie-setup.sh prepare-war
or by
addtowar.sh -extjs EXTJS_PATH
Make sure that the ext.js file was present in the LIBEXT in the oozie home while using preparewar.
Reference:
https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Server_Setup
Related
I am getting below error while executing sqoop export command(in shell script) with oozie.
"java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver"
sqoop export from cli(edge node) works fine.
I have added the ojdbc6.jar to below locations.
/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/sqoop/lib/
(HDFS locations)
/user/oozie/share/lib/sqoop/ and
/user/oozie/share/lib/lib_20161215195933/sqoop
i have also set oozie.use.system.libpath=true in my oozie job.properties file
Please guide me if i am missing any setting.
log content
Thanks & Regards,
Sonali
Make sure that you upload a file to a directory /user/oozie/share/lib/sqoop (it could looks like /user/oozie/share/lib/lib_${timestamp}/sqoop for Cloudera and HDP).
Check if ojdbc6.jar file is correct - check if it contains OracleDriver.class and make sure size of the file is ok. It could be error while downloading.
Check permissions to ojdbc6.jar file (eventually, you can try to give 755 permissions to this file). Check who is the owner of the file - it should be oozie by default.
Update Oozie sharelib by execute below command (run this command on the host where Oozie Server is located):
sudo -u oozie oozie admin -oozie http://<Oozie_Server_Host>:11000/oozie -sharelibupdate
Verify sharelib for sqoop:
sudo -u oozie oozie admin -oozie http://<Oozie_Server_Host>:11000/oozie -shareliblist sqoop*
You can always restart Oozie service. It should update sharelib.
Create a directory named lib next to your workflow.xml in HDFS and put jars in there. Oozie will automatically make those jars available to all actions in that workflow.
Cloudera users should check this article. Especially paragraph 'One Last Thing'.
I'm trying to add hive-contrib-0.10.0.jar in hive using ADD JAR hive-contrib-0.10.0.jar command but it always saying hive-contrib-0.10.0.jar does not exist.
I'm using HDP 2.1 version right now. I also added this jar file into /user/root folder using hue and run the command
ADD JAR hdfs:///hive-contrib-0.10.0.jar
but it giving me same error jar file doesn't exist.
Is there any way to solve this problem.
Where should I keep this jar file so that it will run successfully and what will be the command to be used?
upload the JAR file into hdfs path
Add the JAR File using Add command and HDFS full PATH
Example:
hadoop fs -put ~/Downloads/hive.jar /lib/
open hive shell
add jar hdfs:///lib/hive.jar
I see following issues with your approach. Before adding make sure you are able to list the file on Local file system or hdfs where ever it exists.
The jar you are trying to add is by default in hive class path as its part of $HIVE_HOME/lib (on local file system where ever you have hive client/service installed)
on the other hand in regards to your question about how to add jars in hive, we can add using local file system or hadoop distributed file system (HDFS)
Add jar file:///root/hive-contrib-0.10.0.jar (Given that you copied this jar on LFS root directory)
Add jar hdfs://<namenode_hostname>:8020/user/root/hive-contrib-0.10.0.jar (Given that you copied to HDFS root home)
if you want to permanently add the jars you need to do the following.
1. Hive-site.xml ( /etc/hive/conf )
<property>
<name>hive.aux.jars.path</name>
<value>file:///mnt1/hive-jars/hive-contrib-2.1.1.jar</value>
</property>
add hive-contrib-2.1.1.jar to the path "/mnt1/hive-jars" configured in hive-site.xml
This should ideally work after restarting hive-server2.
3. sudo stop hive-server2
4. sudo start hive-server2
But sometimes it does not work. i am not sure why so you can use the following dirty way.
put your jar file in the following path so that hive automatically picks it up while restart.
add hive-contrib-2.1.1.jar to /usr/lib/hive-hcatalog/share/hcatalog
sudo stop hive-server2
sudo start hive-server2
I have read these answers above which was very useful. And I combined all into one solution:
put jars into local disk and give read/write permission
chmod -R 777 /tmp/json.jar
upload to hdfs file system and give permissions too:
hdfs dfs -put /tmp/json.jar hdfs://1.1.1.1:8020/jars/
hdfs dfs -chmod -R 777 hdfs://1.1.1.1:8020/jars/
add jar into hive env.
add jar hdfs://1.1.1.1:8020/jars/json.jar
You have to give the full path to the jar JAR and not only its name.
Don't guess the location. Check the file system to see that it is there, before trying to add it.
I am setting up oozie for first time. After all the setup is done, when I ran example workflows, they failed. And -errorlog shows following message
org.apache.oozie.action.ActionExecutorException:
File /user/userName/share/lib does not exist
I did run the below command to create sharelib while doing setup and I can see the sharelib with all the jars in hdfs webconsole-
./oozie-setup.sh sharelib create -fs hdfs://localhost:9000
but when I verify the available sharelib using
oozie admin -shareliblist
it shows only [Availble Sharelib] message with no libs actually listed.
Any idea, what can be the issue?
I am a new pyspark user.
I just downloaded and installed a spark cluster ("spark-2.0.2-bin-hadoop2.7.tgz")
after installation I wanted to access the file system (upload local files to cluster). But when I tried to type hadoop or hdfs in command it will say "no command found".
Am I gonna install hadoop/HDFS (I thought it's built in the spark, I don't get)?
Thanks in advance.
You have to install hadoop first to access HDFS.
Follow this http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/
Choose the latest version of hadoop from the apache site.
Once you done with hadoop setup go to spark http://d3kbcqa49mib13.cloudfront.net/spark-2.0.2-bin-hadoop2.7.tgz download this, Extract files. Setup java_home and hadoop_home in spark-env.sh.
You don't have hdfs or hadoop on classpath so this is the reason why you are getting message: "no command found".
If you run \yourparh\hadoop-2.7.1\bin\hdfs dfs -ls / it should works and show root content.
But, You can add your hadoop/bin (hdfs, hadoop ...) commands to classpath with something like this:
export PATH $PATH:$HADOOP_HOME/bin
where HADOOP_HOME is your env. variable with path to hadoop installation folder (download and install is required)
The details of my oozie job are given below:
I am getting the following error:
57-oozie-oozi-W#MR] Error starting action [MR]. ErrorType [FAILED], ErrorCode [EJ001], Message [Could not locate Oozie sharelib]
org.apache.oozie.action.ActionExecutorException: Could not locate Oozie sharelib
at org.apache.oozie.action.hadoop.JavaActionExecutor.addSystemShareLibForAction(JavaActionExecutor.java:603)
at org.apache.oozie.action.hadoop.JavaActionExecutor.addAllShareLibs(JavaActionExecutor.java:698)
at org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:689)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:884)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1135)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:228)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
I have put lib in /user/root ,user/oozie .But still it is not able to find it.
Quoting the Oozie documentation a MapReduce action should not care about Shared Libs:
The Oozie sharelib TAR.GZ file bundled with the distribution contains
the necessary files to run Oozie map-reduce streaming, pig, hive,
sqooop, and distcp actions ... other actions (mapreduce, shell, ssh,
and java) do not require the sharelib to be installed.
Anyway, the root dir for the Shared Libs should be something like
/user/oozie/share/lib/
Your setup is probably missing the "share" part (see Quick Start)
And if you want to try a non-default location then look at the documentation for "sharelib create" command, and also for specifics about Oozie server config.
http://oozie.apache.org/docs/4.1.0/AG_Install.html#Oozie_Share_Lib
plus #Oozie_Server_Setup