I want to remove oozie and reinstall a fresh copy.
I installed oozie by following this steps
http://hadooptutorial.info/apache-oozie-installation-on-ubuntu-14-04/ Can anyone please help me to remove oozie completely from my laptop?
I am using ubuntu latest version ..with hadoop 2.6.0 ..
Earlier I removed /usr/lib/oozie folder but it did not worked out for me after installing a fresh copy of oozie ..(got many errors and exception )
I am describing few of the errrors below after installing fresh copy of oozie
oozie admin -oozie http://localhost:11000/oozie -status
Connection exception has occurred [ java.net.ConnectException Connection refused ]. Trying after 1 sec. Retry count = 1
oozied.sh stop
PID file found but no matching process was found. Stop aborted.
oozie-setup.sh sharelib create -fs hdfs://localhost:9000
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/io/filefilter/IOFileFilter
Thank you
removing /usr/lib/oozie will not remove oozie entirely .
Something more is require
Related
Once I install Hadoop and type hdfs namenode –format or hadoop namenode -format in cmd for the 1st time,
Am getting below error, can anyone help me in solving this.
1st it is asking me this:
Re-format filesystem in Storage Directory root= C:\hadoop-3.2.1\data\namenode; location= null ? (Y or N)
No matter what I give i.e., Y or N, am getting the below error.
ERROR namenode.NameNode: Failed to start namenode
ERROR namenode.NameNode: Failed to start namenode.
java.lang.UnsupportedOperationException
INFO util.ExitUtil: Exiting with status 1: java.lang.UnsupportedOperationException
Quick answer is much appreciated
Regards
ShaX
This is a bug in 3.2.1 release and is supposed to fixed in 3.2.2 or 3.3.0.
The fix is to change the StorageDirectory class by adding FileUtil for Windows permission setup:
if (permission != null) {
try {
Set<PosixFilePermission> permissions =
PosixFilePermissions.fromString(permission.toString());
Files.setPosixFilePermissions(curDir.toPath(), permissions);
} catch (UnsupportedOperationException uoe) {
// Default to FileUtil for non posix file systems
FileUtil.setPermission(curDir, permission);
}
}
I found this issue when publishing a Hadoop 3.2.1 installation guide on Windows:
Latest Hadoop 3.2.1 Installation on Windows 10 Step by Step Guide
I published a temporary resolution and it is working. Refer to my above post for details and you can follow it to complete Hadoop 3.2.1 installation on Windows 10. I've uploaded my updated Hadoop HDFS jar file to the following location:
https://github.com/FahaoTang/big-data/blob/master/hadoop-hdfs-3.2.1.jar
I was running Spark job which make use of other files that is passed in through --archives flag of spark
spark-submit .... --archives hdfs:///user/{USER}/{some_folder}.zip .... {file_to_run}.py
Spark is currently running on YARN and when I tried it with spark version 1.5.1 it was fine.
However, when I ran the same commands with spark 2.0.1, I got
ERROR yarn.ApplicationMaster: User class threw exception: java.io.IOException: Cannot run program "/home/{USER}/{some_folder}/.....": error=2, No such file or directory
Since the resource is managed by YARN, it is challenging to manually check if the file gets successfully decompressed and exist when the job runs.
I wonder if anyone has experienced similar issue.
I am new to oozie, trying to write a oozie workflow in CDH4.1.1. So I started the oozie service and then I checked the status using this command:
sudo service oozie status
I got the message:
running
Then I tried this command for checking the status:
oozie admin --oozie http://localhost:11000/oozie status
And I got the below exception:
java.lang.NullPointerException
at java.io.Writer.write(Writer.java:140)
at org.apache.oozie.client.AuthOozieClient.writeAuthToken(AuthOozieClient.java:182)
at org.apache.oozie.client.AuthOozieClient.createConnection(AuthOozieClient.java:137)
at org.apache.oozie.client.OozieClient.validateWSVersion(OozieClient.java:243)
at org.apache.oozie.client.OozieClient.createURL(OozieClient.java:344)
at org.apache.oozie.client.OozieClient.access$000(OozieClient.java:76)
at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:410)
at org.apache.oozie.client.OozieClient.getSystemMode(OozieClient.java:1299)
at org.apache.oozie.cli.OozieCLI.adminCommand(OozieCLI.java:1323)
at org.apache.oozie.cli.OozieCLI.processCommand(OozieCLI.java:499)
at org.apache.oozie.cli.OozieCLI.run(OozieCLI.java:466)
at org.apache.oozie.cli.OozieCLI.main(OozieCLI.java:176)
null
Reading the exception stack, I am unable to figure out the reason for this exception. Please let me know why I got this exception and how to resolve this.
Try disabling the env property USE_AUTH_TOKEN_CACHE_SYS_PROP in your cluster. As per your stacktrace and the code .
Usually the clusters are setup with Kerberos based authentication, which is set up by following the steps here . Not sure if you want to do that, but just wanted to mentioned that as an FYI.
I can run hadoop jar apache-nutch-1.7.job org.apache.nutch.crawl.Crawl args in command line just fine, but when run in oozie, I get an exception
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File does not exist: hdfs://server:8020/user/hdfs/.staging/job_1416525929767_0494/job.splitmetainfo
at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.createSplits(JobImpl.java:1566)
at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1430)
at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1388)
an old Jira reported this exception
https://issues.apache.org/jira/browse/MAPREDUCE-5471
but it was supposely fixed back in version 2.1.1-beta. I am on yarn 2.5.0.
Any one else see this?
Job running in oozie is getting the following error:
hduser#ubuntu:~/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2$ bin/oozie job -oozie -config examples/apps/map-reduce/job.properties -run
Error: E0902 : E0902: Exception occured: [Call to 127.0.0.1:8020 failed on local exception: java.io.IOException: Broken pipe]
How can I solve this. Thanks
Looks like Oozie has trouble connecting to your name-node.
Its trying to connect to 127.0.0.1:8020, is this where the namenode is running?
If yes, double check it is running.
If no, make sure job.properties has a line like: nameNode=hdfs://namenode_host:8020
Which should point to the correct name node location.
Reference ${nameNode} in the section of your action.