My coordinator failed with Error : E0301 invalid resource [filename]
when I do hadoop fs -ls [filename] the file is listed.
how can I debug what is wrong.
how can I check log files???
oozie job -log requires jobId. in my case i dont have job id. how can I see logs in that case. appreciate responses.
thank you
If you are looking for a command line way to do this, you can run the following:
oozie job -oozie http://localhost:11000 -info <wfid>
oozie job -oozie http://localhost:11000 -log <wfid>
If you have the $OOZIE_URL set, then you do not need the -oozie parm in the above statements. This first command will show you the status of the job and each action. The second command will dig into the oozie log and display the part in the log that pertains to the workflow id that was passed in.
cd /var/log/oozie/
ls
You should see the log file there.
I highly recommend using the oozie webconsole when new to oozie. If you are using Cloudera it's under "Enabling the Oozie Web Console" here http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_17_6.html for CDH4. CDH3 link is similar.
Also the jobid is printed when you submit the job.
Related
Can you please someone help me to get the oozie error logs to hive table when jobs get failed. suggest me the approach how to do as i am new to this.
Write a shell script to pick up oozie job logs using:
oozie job -oozie http://localhost:11000 -info <wfid>
oozie job -oozie http://localhost:11000 -log <wfid>
redirect results of it into a file, which you may load into a hive table. Then use a oozie shell action trigger this step at failure.
as a newbie with Oozie, I tried to run some tutorials by command line. My stepByStep:
upload my Oozie project (workflow xml file, job.properties file, jar and data) to HDFS via HUE interface. In my job.properties files, I've indicated every information like data name node, path to my application, ...
running via HUE interface, simply, I check on check box of workflow xml file and submit
I would like to run my Oozie project by command line:
with job.properties file in local, I run:
oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run
How can I run my Oozie project instead of with the job.properties in local (instead of the config file in local, I want to run my job with the configuration file in HDFS)?
Thanks for any suggestion and feel free to comment if my question is not clear!
I don't know if there is a direct way, but you certainly could do something like
oozie job -oozie http://localhost:11000/oozie -config <(hdfs dfs -cat examples/apps/map-reduce/job.properties) -run
I tried running my first oozie job by following a blog post.
I used oozie-examples.tar.gz, after extracting, placed examples in hdfs.
I tried running map-reduce job in it but unfortunately got an error.
Ran below command:
oozie job -oozie http://localhost:11000/oozie -config /examples/apps/map-reduce/job.properties -run
Got the error:
java.io.IOException: configuration is not specified at
org.apache.oozie.cli.OozieCLI.getConfiguration(OozieCLI.java:787) at
org.apache.oozie.cli.OozieCLI.jobCommand(OozieCLI.java:1026) at
org.apache.oozie.cli.OozieCLI.processCommand(OozieCLI.java:662) at
org.apache.oozie.cli.OozieCLI.run(OozieCLI.java:615) at
org.apache.oozie.cli.OozieCLI.main(OozieCLI.java:218) configuration is
not specified
I don't know which configuration it is asking for as I am using Cloudera VM and it has by default got all the configurations set in it.
oozie job -oozie http://localhost:11000/oozie -config /examples/apps/map-reduce/job.properties -run
The -config parameter takes an local path not an HDFS path. The workflow.xml needs to be present in the HDFS and path is defined in the job.properties file with the property:
oozie.wf.application.path=<path to the workflow.xml>
I have checked for oozie service at 11000. It is connecting.
But at the time of submitting job console is stuck.
Command used for submitting is
oozie/bin/oozie job -submit -config /tmp/config.properties -oozie http://127.0.0.1:11000/oozie
I have also checked logs for errors. There isnt any.
You are using the -submit command. You need to use -run for the job to submit. You should get a workflow id when you submit. This is the command you should run:
oozie/bin/oozie job -run -config /tmp/config.properties -oozie http://127.0.0.1:11000/oozie
You can check the status of the job by running:
oozie job -oozie http://localhost:11000/oozie -info <wfid>
I have mentioned wrong yarn port and ip address in config file. Thats why it was not connecting to yarn and it was not submitting. I have updated it and its working fine.
I am using Oozie 4.2 that comes bundled with HDP 2.3.
while working with a few example workflow's that comes with the oozie package, I noticed that the "job error log is disabled" and this makes debugging really difficult in the event of a failure. I tried running the below commands,
# oozie job -config /home/santhosh/examples/apps/hive/job.properties -run
job: 0000063-150904123805993-oozie-oozi-W
# oozie job -errorlog 0000063-150904123805993-oozie-oozi-W
Error Log is disabled!!
Can someone please tell me how to enable the workflow error log for oozie?
In the Oozie UI, 'Job Error Log' is a tab which was introduced in HDP v2.3 on Oozie v4.2 .
This is the most simplest way of looking for error for the specified oozie job from the oozie log file.
To enable the oozie's Job Error Log, please make the following changes in the oozie log4j property file:
Add the below set of lines after log4j.appender.oozie and before log4j.appender.oozieops:
log4j.appender.oozieError=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.oozieError.RollingPolicy=org.apache.oozie.util.OozieRollingPolicy
log4j.appender.oozieError.File=${oozie.log.dir}/oozie-error.log
log4j.appender.oozieError.Append=true
log4j.appender.oozieError.layout=org.apache.log4j.PatternLayout
log4j.appender.oozieError.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - SERVER[${oozie.instance.id}] %m%n
log4j.appender.oozieError.RollingPolicy.FileNamePattern=${log4j.appender.oozieError.File}-%d{yyyy-MM-dd-HH}
log4j.appender.oozieError.RollingPolicy.MaxHistory=720
log4j.appender.oozieError.filter.1 = org.apache.log4j.varia.LevelMatchFilter
log4j.appender.oozieError.filter.1.levelToMatch = WARN
log4j.appender.oozieError.filter.2 = org.apache.log4j.varia.LevelMatchFilter
log4j.appender.oozieError.filter.2.levelToMatch = ERROR
log4j.appender.oozieError.filter.3 =`enter code here` org.apache.log4j.varia.LevelMatchFilter
log4j.appender.oozieError.filter.3.levelToMatch = FATAL
log4j.appender.oozieError.filter.4 = org.apache.log4j.varia.DenyAllFilter
Modify the following from log4j.logger.org.apache.oozie=WARN, oozie to log4j.logger.org.apache.oozie=ALL, oozie, oozieError
Restart the oozie service. This would help in getting the job error log for the new jobs launched after restart of oozie service.
As mentioned, the errorlog is new and may not be made available for good reasons. However it seems that you have the wrong expectation of the oozie error log.
The error log is meant to be a subset of the log file. Not an addition to it.
So yes, it could make things easier to debug, but if you checked the oozie log and did not find what you are looking for, the error log will not be the solution for you.
Probably you will want to look at the log of the underlying tasks, which can be found via the external ID.