Here is my workflow.xml
<action name="hive-node">
<hive xmlns="uri:oozie:hive-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/user/${wf:user()}/${wfeRoot}/output-data/hive"/>
<mkdir path="${nameNode}/user/${wf:user()}/${wfeRoot}/output-data"/>
</prepare>
<job-xml>hive-site.xml</job-xml>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
<property>
<name>oozie.log.hive.level</name>
<value>DEBUG</value>
</property>
<property>
<name>oozie.hive.defaults</name>
<value>hive-default.xml</value>
</property>
</configuration>
<script>script.q</script>
</hive>
<ok to="end"/>
<error to="fail"/>
Hive failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
my job.properties file
nameNode=hdfs://localhost:8020
jobTracker=localhost:8021
queueName=default
wfeRoot=wfe
oozie.use.system.libpath=true
oozie.libpath=/user/oozie/share/lib/hive
oozie.wf.application.path=${nameNode}/user/${user.name}/${wfeRoot}/hiveoozie
Script
create table brundesh(name string,lname string) row format delimited fields terminated by ',';
I copied hive-site.xml ,script.hql and hive-default.xml in to oozie app directory. I am using cdh3
Error detalis:
Error code: JA018
Error Message: Main Class[org.apache.oozie.action.hadoop.HiveMain],exit code [9]
I copied the required jar files to sharelib directory in hdfs. I copied all the jar fiels present in oozie.sharelib.tar.gz from $OOZIE_HOME
I goggled for error but no luck. Please help me were am going wrong
As mention by Ben Please check Hive Log, which present in the respected Node or Check with in the console URL with details of the Logs.
Will also suggest to do another steps which requried to perform are:
Take a Backup of Shared Lib Jar from the DFS Location.
Upload the same Jars from local Hive Lib Location to DFS Shared Location with Oozie User.
Make Sure there should not be any Duplicate Hive Jar present in other Local Location except Hive Lib Path.
All Nods should be having the same Jars.
If you are using Pig as well, then please perform the Step 1, Step 2 , Step 3 from Pig as well.
Check the Hadoop ClassPath if there Classpath have been set properly.
Related
I have python script which I 'm able to run through spark-submit. I need to use it in Oozie.
<!-- move files from local disk to hdfs -->
<action name="forceLoadFromLocal2hdfs">
<shell xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<exec>driver-script.sh</exec>
<!-- single -->
<argument>s</argument>
<!-- py script -->
<argument>load_local_2_hdfs.py</argument>
<!-- local file to be moved-->
<argument>localPathFile</argument>
<!-- hdfs destination folder, be aware of, script is deleting existing folder! -->
<argument>hdfFolder</argument>
<file>${workflowRoot}driver-script.sh#driver-script.sh</file>
<file>${workflowRoot}load_local_2_hdfs.py#load_local_2_hdfs.py</file>
</shell>
<ok to="end"/>
<error to="killAction"/>
</action>
The script by itself through driver-script.sh runs fine. Through oozie, even the status of workflow is SUCCEEDED, the file is not copied to hdfs. I was not able to find any error logs, or related logs to pyspark job.
I have another topic about supressed logs from Spark by oozie here
Set your script to set -x in the beginning that will show you which line the script is it. You can see those in the stderr.
Can you elaborate on what you mean by file is not copied ? To help you better.
I am able to run a sqoop command in Oozie using Hue. But, when I try to run the same sqoop command by placing it in a shell script I am getting an error like below
Stdoutput 2016-05-20 10:52:13,241 ERROR [main] sqoop.Sqoop (Sqoop.java:runSqoop(181)) - Got exception running Sqoop:
java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
I have included the jdbc jar file like I did while running the sqoop command directly. I don't understand why it is not working for shell script.
Here is the workflow generated by Hue
<workflow-app name="My_Workflow" xmlns="uri:oozie:workflow:0.5">
<start to="shell-ca31"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="shell-ca31">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>default</value>
</property>
<property>
<name>oozie.use.system.libpath</name>
<value>true</value>
</property>
<property>
<name>oozie.libpath</name>
<value>/user/oozie/libext</value>
</property>
</configuration>
<exec>sqoopoozie.sh</exec>
<file>/user/yxr6907/sqoopoozie.sh#sqoopoozie.sh</file>
<archive>/user/oozie/libext/ojdbc7.jar#ojdbc7.jar</archive>
<capture-output/>
</shell>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>
When you use shell action, jars for sqoop are not imported into classpath.
I was able to solve it by adding the jar into the classpath. Then, i export HADOOP_CLASSPATH and sqoop works.
Use the following:
Put the jar ojdbc7.jar in files
Use the following command inside shell script: export HADOOP_CLASSPATH=${PWD}/ojdbc7.jar
Instead of step 1. you can use the following properties to load jar into classpath:
oozie.use.system.libpath=true
oozie.libpath=/path/to/jars
Exporting HADOOP_CLASSPATH is required in both ways.
I am trying to do very basic oozie workflow
I am getting the below error wheni give the command..
user#ubuntu:~/surender$ oozie job -oozie http://localhost:11000/oozie /home/user/surender/oozie_demo/job.properties -run
Error:
Error: E0501 : E0501: Could not perform authorization operation, Failed on local exception: java.io.EOFException; Host Details : local host is: "ubuntu/127.0.0.1"; destination host is: "localhost":8020;
My oozie version is 4.0.0 , I checked that oozie web console is enabled..
This is how created a oozie workflow
I created a directory called oozie_demo and inside that i created two files
1.workflow.xml
2.job.properties
I also created a lib directory and placed the pig script inside that lib directory
workflow.xml
<workflow-app xmlns="uri:oozie:workflow:0.2" name="pig-wf">
<start to="pig-node"/>
<action name="pig-node">
<pig>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/user/${wf:user()}/output/pig/simple_load"/>
</prepare>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
</configuration>
<script>simple_load.pig</script>
<param>INPUT=/user/${wf:user()}/inputfiles/records.txt</param>
<param>OUTPUT=/user/${wf:user()}//output/pig/simple_load</param>
</pig>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Pig failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
job.properties
nameNode=hdfs://localhost:8020
jobTracker=localhost:8021
queueName=default
oozie_demo=oozie_demo
oozie.use.system.libpath=true
ozie.wf.application.path=${nameNode}/user/user/oozie_demo
my pig script :
records = load '/user/user/inputfiles/records.txt' USING PigStorage(',');
store records into '/user/user/output/pig/simple_load' using PigStorage(',');
Could somebody help me on this? I would like to know what went wrong? and how do i resolve this issue ?
Could you check if the Namenode is up and running at port 8020.
I'm trying to do a simple job using oozie. It will be a one simple Pig Action.
I have a file : FirstScript.pig containing:
dual = LOAD 'default.dual' USING org.apache.hcatalog.pig.HCatLoader();
store dual into 'dummy_file.txt' using PigStorage();
and a workflow.xml containing:
<workflow-app name="FirstWorkFlow" xmlns="uri:oozie:workflow:0.2">
<start to="FirstJob"/>
<action name="FirstJob">
<pig>
<job-tracker>hadoop:50300</job-tracker>
<name-node>hdfs://hadoop:8020</name-node>
<script>/FirstScript.pig</script>
</pig>
<ok to="okjob"/>
<error to="errorjob"/>
</action>
<ok name='okjob'>
<message>job OK, message[${wf:errorMessage()}]</message>
</ok>
<error name='errorjob'>
<message>job error, error message[${wf:errorMessage()}]</message>
</error>
</workflow-app>
I have created structure :
FirstScript
|- lib
|---FirstScript.pig
|- workflow.xml
And what now?
How do I deploy it and run with oozie?
Can anyone more experienced help?
Regards
Pawel
I do it like this:
hadoop fs -put workflow.xml some_dir/
oozie job --oozie http://your_host:11000/oozie -config cluster_conf.xml -run
and my cluster_conf.xml looks like this (please check your ports first they depend on Hadoop distro):
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<configuration>
<property>
<name>nameNode</name>
<value>hdfs://my_nn:8020</value>
</property>
<property>
<name>jobTracker</name>
<value>my_jt:8050</value>
</property>
<property>
<name>oozie.wf.application.path</name>
<value>/user/my_user/some_dir/workflow.xml</value>
</property>
</configuration>
Config file should point to job.properties in place of file.xml. Since, job.properties contains path to workflow.xml
oozie job --oozie http://your_host:11000/oozie -config **/job.properties** -run
I'm trying to run a hive action through Oozie. My workflow.xml is as follows:
<workflow-app name='edu-apollogrp-dfe' xmlns="uri:oozie:workflow:0.1">
<start to="HiveEvent"/>
<action name="HiveEvent">
<hive xmlns="uri:oozie:hive-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>oozie.hive.defaults</name>
<value>${hiveConfigDefaultXml}</value>
</property>
</configuration>
<script>${hiveQuery}</script>
<param>OUTPUT=${StagingDir}</param>
</hive>
<ok to="end"/>
<error to="end"/>
</action>
<kill name='kill'>
<message>Hive failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name='end'/>
And here is my job.properties file:
oozie.wf.application.path=${nameNode}/user/${user.name}/hiveQuery
oozie.libpath=${nameNode}/user/${user.name}/hiveQuery/lib
queueName=interactive
#QA
nameNode=hdfs://hdfs.bravo.hadoop.apollogrp.edu
jobTracker=mapred.bravo.hadoop.apollogrp.edu:8021
# Hive
hiveConfigDefaultXml=/etc/hive/conf/hive-default.xml
hiveQuery=hiveQuery.hql
StagingDir=${nameNode}/user/${user.name}/hiveQuery/Output
When I run this workflow, I end up with this error:
ACTION[0126944-130726213131121-oozie-oozi-W#HiveEvent] Launcher exception: org/apache/hadoop/hive/cli/CliDriver
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/cli/CliDriver
Error Code: JA018
Error Message: org/apache/hadoop/hive/cli/CliDriver
I'm not sure what this error means. Where am I going wrong?
EDIT
This link says error code JA018 is: JA018 is output directory exists error in workflow map-reduce action. But in my case the output directory does not exist. This makes it all the more confusing
I figured out what was going wrong!
The class org/apache/hadoop/hive/cli/CliDriver is required for execution of a Hive Action. This much is obvious from the error message. This class is within this jar file: hive-cli-0.7.1-cdh3u5.jar. (In my case cdh3u5 in my cloudera version).
Oozie checks for this jar in the ShareLib directory. The location of this directory is usually configured in hive-site.xml, with the property name as oozie.service.WorkflowAppService.system.libpath, so Oozie should find the jar easily.
But in my case, hive-site.xml did not include this property, so Oozie didn't know where to look for this jar, hence the java.lang.NoClassDefFoundError.
To resolve this, I had to include a parameter in my job.properties file to point oozie to the location of the ShareLib directory, as follows:
oozie.libpath=${nameNode}/user/oozie/share/lib. (depends on where SharedLib directory is configured on your cluster).
This got rid of the error!