Hadoop MapReduce Job stuck because auxService:mapreduce_shuffle does not exist - hadoop

I've checked multiple posts with the same questions, and the solution is always adding the following to the yarn-site.xml
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarm.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
I covered both properties in the config and restarted yarn. The problem still remains.
The error is:
17/02/15 15:43:34 INFO mapreduce.Job: Task Id : attempt_1487202110321_0001_m_000000_2, Status : FAILED
Container launch failed for container_1487202110321_0001_01_000007 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I was hoping for a typo but can't seem to find it. Tried directly copy what's on stackoverflow into the xml file, still doesn't work.
What else can I try?
EDIT:
Since the error says the aux_service should be auxService, i modified the yarn-site.xml according, changing all aux-service to auxService, but it's still not working.
EDIT2:
In case anyone's interested, I call this command
hadoop jar hadoop-streaming-2.7.1.jar \
-input /user/myfolder/input1/* \
-output /user/myfolder/output1 \
-mapper <path>/<to>/<mapper>/mapper.py \
-reducer <path>/<to>/<reducer>/reducer.py
while I'm already in /usr/local/cellar/hadoop/2.7.1/libexec/share/hadoop/tools/lib/
EDIT 3:
I'm a dumbass. proof-read the script guys!

Update the property name in yarn-site.xml as yarn.nodemanager.aux-services,
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

Related

Hadoop installation issue ubuntu 18.04: hadoop namenode -format error

I'm trying to do a hadoop installation.
I am following this article for hadoop installation instructions. One of the steps I need to do is to format the hadoop file system using the command:
root#ben-Aspire-E5-575G:~# hadoop namenode -format
I got the following error
2018-10-12 00:08:16,884 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2018-10-12 00:08:16,896 INFO namenode.NameNode: createNameNode [-format]
2018-10-12 00:08:17,024 ERROR conf.Configuration: error parsing conf hdfs-site.xml
com.ctc.wstx.exc.WstxEOFException: Unexpected EOF; was expecting a close tag for element <xml>
at [row,col,system-id]: [49,0,"file:/usr/local/hadoop/etc/hadoop/hdfs-site.xml"]
at com.ctc.wstx.sr.StreamScanner.throwUnexpectedEOF(StreamScanner.java:687)
at com.ctc.wstx.sr.BasicStreamReader.throwUnexpectedEOF(BasicStreamReader.java:5608)
at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2802)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3257)
at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3063)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2986)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2926)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1366)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1679)
at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:339)
at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:572)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:174)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:156)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1587)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)
2018-10-12 00:08:17,031 ERROR namenode.NameNode: Failed to start namenode.
java.lang.RuntimeException: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF; was expecting a close tag for element <xml>
at [row,col,system-id]: [49,0,"file:/usr/local/hadoop/etc/hadoop/hdfs-site.xml"]
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3003)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2926)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1366)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1679)
at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:339)
at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:572)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:174)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:156)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1587)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)
Caused by: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF; was expecting a close tag for element <xml>
at [row,col,system-id]: [49,0,"file:/usr/local/hadoop/etc/hadoop/hdfs-site.xml"]
at com.ctc.wstx.sr.StreamScanner.throwUnexpectedEOF(StreamScanner.java:687)
at com.ctc.wstx.sr.BasicStreamReader.throwUnexpectedEOF(BasicStreamReader.java:5608)
at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2802)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3257)
at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3063)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2986)
... 11 more
2018-10-12 00:08:17,035 INFO util.ExitUtil: Exiting with status 1: java.lang.RuntimeException: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF; was expecting a close tag for element <xml>
at [row,col,system-id]: [49,0,"file:/usr/local/hadoop/etc/hadoop/hdfs-site.xml"]
2018-10-12 00:08:17,043 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ben-Aspire-E5-575G/127.0.1.1
************************************************************/
the hdfs-site.xml file is here
<xml version="1.0" encoding="UTF-8">
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create t$
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop_store/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop_store/hdfs/datanode</value>
</property>
</configuration>
You're missing both </description> and a final </xml> outside of the end configuration tag in the file you've provided.
As the error says, there's a tag missing.
The description is optional, so feel free to delete that tag entirely
I think you have to fix the "description" tag. There is no proper end.

Hadoop Single Node Cluster setup error during namenode format

I have installed Apache Hadoop 2.6.0 in Windows 10. I have been trying to fix this issue but failed to understand the error or any mistake from my end.
I have set up all the paths correctly, Hadoop version is showing the version in command prompt properly.
I have already created temp directory inside hadoop directory like c:\hadoop\temp.
When I am trying to format the Namenode, I am getting this error:
C:\hadoop\bin>hdfs namenode -format
18/07/18 20:44:55 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = TheBhaskarDas/192.168.44.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.6.5
STARTUP_MSG: classpath = C:\hadoop\etc\hadoop;C:\hadoop\share\hadoop\common\lib\activation-1.1.jar;C:\hadoop\share\hadoop\common\lib\apacheds-i18n-2.0.0-M15.jar;C:\hadoop\share\hadoop\common\lib\apacheds-kerberos-codec-2.0.0-M15.jar;C:\hadoop\share\hadoop\common\lib\api-asn1-api-1.0.0-M20.jar;C:\hadoop\share\hadoop\common\lib\api-util-1.0.0-M20.jar;C:\hadoop\share\hadoop\common\lib\asm-3.2.jar;C:\hadoop\share\hadoop\common\lib\avro-1.7.4.jar;C:\hadoop\share\hadoop\common\lib\commons-beanutils-1.7.0.jar;C:\hadoop\share\hadoop\common\lib\commons-beanutils-core-1.8.0.jar;C:\hadoop\share\hadoop\common\lib\commons-cli-1.2.jar;C:\hadoop\share\hadoop\common\lib\commons-codec-1.4.jar;C:\hadoop\share\hadoop\common\lib\commons-collections-3.2.2.jar;C:\hadoop\share\hadoop\common\lib\commons-compress-1.4.1.jar;C:\hadoop\share\hadoop\common\lib\commons-configuration-1.6.jar;C:\hadoop\share\hadoop\common\lib\commons-digester-1.8.jar;C:\hadoop\share\hadoop\common\lib\commons-el-1.0.jar;C:\hadoop\share\hadoop\common\lib\commons-httpclient-3.1.jar;C:\hadoop\share\hadoop\common\lib\commons-io-2.4.jar;C:\hadoop\share\hadoop\common\lib\commons-lang-2.6.jar;C:\hadoop\share\hadoop\common\lib\commons-logging-1.1.3.jar;C:\hadoop\share\hadoop\common\lib\commons-math3-3.1.1.jar;C:\hadoop\share\hadoop\common\lib\commons-net-3.1.jar;C:\hadoop\share\hadoop\common\lib\curator-client-2.6.0.jar;C:\hadoop\share\hadoop\common\lib\curator-framework-2.6.0.jar;C:\hadoop\share\hadoop\common\lib\curator-recipes-2.6.0.jar;C:\hadoop\share\hadoop\common\lib\gson-2.2.4.jar;C:\hadoop\share\hadoop\common\lib\guava-11.0.2.jar;C:\hadoop\share\hadoop\common\lib\hadoop-annotations-2.6.5.jar;C:\hadoop\share\hadoop\common\lib\hadoop-auth-2.6.5.jar;C:\hadoop\share\hadoop\common\lib\hamcrest-core-1.3.jar;C:\hadoop\share\hadoop\common\lib\htrace-core-3.0.4.jar;C:\hadoop\share\hadoop\common\lib\httpclient-4.2.5.jar;C:\hadoop\share\hadoop\common\lib\httpcore-4.2.5.jar;C:\hadoop\share\hadoop\common\lib\jackson-core-asl-1.9.13.jar;C:\hadoop\share\hadoop\common\lib\jackson-jaxrs-1.9.13.jar;C:\hadoop\share\hadoop\common\lib\jackson-mapper-asl-1.9.13.jar;C:\hadoop\share\hadoop\common\lib\jackson-xc-1.9.13.jar;C:\hadoop\share\hadoop\common\lib\jasper-compiler-5.5.23.jar;C:\hadoop\share\hadoop\common\lib\jasper-runtime-5.5.23.jar;C:\hadoop\share\hadoop\common\lib\java-xmlbuilder-0.4.jar;C:\hadoop\share\hadoop\common\lib\jaxb-api-2.2.2.jar;C:\hadoop\share\hadoop\common\lib\jaxb-impl-2.2.3-1.jar;C:\hadoop\share\hadoop\common\lib\jersey-core-1.9.jar;C:\hadoop\share\hadoop\common\lib\jersey-json-1.9.jar;C:\hadoop\share\hadoop\common\lib\jersey-server-1.9.jar;C:\hadoop\share\hadoop\common\lib\jets3t-0.9.0.jar;C:\hadoop\share\hadoop\common\lib\jettison-1.1.jar;C:\hadoop\share\hadoop\common\lib\jetty-6.1.26.jar;C:\hadoop\share\hadoop\common\lib\jetty-util-6.1.26.jar;C:\hadoop\share\hadoop\common\lib\jsch-0.1.42.jar;C:\hadoop\share\hadoop\common\lib\jsp-api-2.1.jar;C:\hadoop\share\hadoop\common\lib\jsr305-1.3.9.jar;C:\hadoop\share\hadoop\common\lib\junit-4.11.jar;C:\hadoop\share\hadoop\common\lib\log4j-1.2.17.jar;C:\hadoop\share\hadoop\common\lib\mockito-all-1.8.5.jar;C:\hadoop\share\hadoop\common\lib\netty-3.6.2.Final.jar;C:\hadoop\share\hadoop\common\lib\paranamer-2.3.jar;C:\hadoop\share\hadoop\common\lib\protobuf-java-2.5.0.jar;C:\hadoop\share\hadoop\common\lib\servlet-api-2.5.jar;C:\hadoop\share\hadoop\common\lib\slf4j-api-1.7.5.jar;C:\hadoop\share\hadoop\common\lib\slf4j-log4j12-1.7.5.jar;C:\hadoop\share\hadoop\common\lib\snappy-java-1.0.4.1.jar;C:\hadoop\share\hadoop\common\lib\stax-api-1.0-2.jar;C:\hadoop\share\hadoop\common\lib\xmlenc-0.52.jar;C:\hadoop\share\hadoop\common\lib\xz-1.0.jar;C:\hadoop\share\hadoop\common\lib\zookeeper-3.4.6.jar;C:\hadoop\share\hadoop\common\hadoop-common-2.6.5-tests.jar;C:\hadoop\share\hadoop\common\hadoop-common-2.6.5.jar;C:\hadoop\share\hadoop\common\hadoop-nfs-2.6.5.jar;C:\hadoop\share\hadoop\hdfs;C:\hadoop\share\hadoop\hdfs\lib\asm-3.2.jar;C:\hadoop\share\hadoop\hdfs\lib\commons-cli-1.2.jar;C:\hadoop\share\hadoop\hdfs\lib\commons-codec-1.4.jar;C:\hadoop\share\hadoop\hdfs\lib\commons-daemon-1.0.13.jar;C:\hadoop\share\hadoop\hdfs\lib\commons-el-1.0.jar;C:\hadoop\share\hadoop\hdfs\lib\commons-io-2.4.jar;C:\hadoop\share\hadoop\hdfs\lib\commons-lang-2.6.jar;C:\hadoop\share\hadoop\hdfs\lib\commons-logging-1.1.3.jar;C:\hadoop\share\hadoop\hdfs\lib\guava-11.0.2.jar;C:\hadoop\share\hadoop\hdfs\lib\htrace-core-3.0.4.jar;C:\hadoop\share\hadoop\hdfs\lib\jackson-core-asl-1.9.13.jar;C:\hadoop\share\hadoop\hdfs\lib\jackson-mapper-asl-1.9.13.jar;C:\hadoop\share\hadoop\hdfs\lib\jasper-runtime-5.5.23.jar;C:\hadoop\share\hadoop\hdfs\lib\jersey-core-1.9.jar;C:\hadoop\share\hadoop\hdfs\lib\jersey-server-1.9.jar;C:\hadoop\share\hadoop\hdfs\lib\jetty-6.1.26.jar;C:\hadoop\share\hadoop\hdfs\lib\jetty-util-6.1.26.jar;C:\hadoop\share\hadoop\hdfs\lib\jsp-api-2.1.jar;C:\hadoop\share\hadoop\hdfs\lib\jsr305-1.3.9.jar;C:\hadoop\share\hadoop\hdfs\lib\log4j-1.2.17.jar;C:\hadoop\share\hadoop\hdfs\lib\netty-3.6.2.Final.jar;C:\hadoop\share\hadoop\hdfs\lib\protobuf-java-2.5.0.jar;C:\hadoop\share\hadoop\hdfs\lib\servlet-api-2.5.jar;C:\hadoop\share\hadoop\hdfs\lib\xercesImpl-2.9.1.jar;C:\hadoop\share\hadoop\hdfs\lib\xml-apis-1.3.04.jar;C:\hadoop\share\hadoop\hdfs\lib\xmlenc-0.52.jar;C:\hadoop\share\hadoop\hdfs\hadoop-hdfs-2.6.5-tests.jar;C:\hadoop\share\hadoop\hdfs\hadoop-hdfs-2.6.5.jar;C:\hadoop\share\hadoop\hdfs\hadoop-hdfs-nfs-2.6.5.jar;C:\hadoop\share\hadoop\yarn\lib\activation-1.1.jar;C:\hadoop\share\hadoop\yarn\lib\aopalliance-1.0.jar;C:\hadoop\share\hadoop\yarn\lib\asm-3.2.jar;C:\hadoop\share\hadoop\yarn\lib\commons-cli-1.2.jar;C:\hadoop\share\hadoop\yarn\lib\commons-codec-1.4.jar;C:\hadoop\share\hadoop\yarn\lib\commons-collections-3.2.2.jar;C:\hadoop\share\hadoop\yarn\lib\commons-compress-1.4.1.jar;C:\hadoop\share\hadoop\yarn\lib\commons-httpclient-3.1.jar;C:\hadoop\share\hadoop\yarn\lib\commons-io-2.4.jar;C:\hadoop\share\hadoop\yarn\lib\commons-lang-2.6.jar;C:\hadoop\share\hadoop\yarn\lib\commons-logging-1.1.3.jar;C:\hadoop\share\hadoop\yarn\lib\guava-11.0.2.jar;C:\hadoop\share\hadoop\yarn\lib\guice-3.0.jar;C:\hadoop\share\hadoop\yarn\lib\guice-servlet-3.0.jar;C:\hadoop\share\hadoop\yarn\lib\jackson-core-asl-1.9.13.jar;C:\hadoop\share\hadoop\yarn\lib\jackson-jaxrs-1.9.13.jar;C:\hadoop\share\hadoop\yarn\lib\jackson-mapper-asl-1.9.13.jar;C:\hadoop\share\hadoop\yarn\lib\jackson-xc-1.9.13.jar;C:\hadoop\share\hadoop\yarn\lib\javax.inject-1.jar;C:\hadoop\share\hadoop\yarn\lib\jaxb-api-2.2.2.jar;C:\hadoop\share\hadoop\yarn\lib\jaxb-impl-2.2.3-1.jar;C:\hadoop\share\hadoop\yarn\lib\jersey-client-1.9.jar;C:\hadoop\share\hadoop\yarn\lib\jersey-core-1.9.jar;C:\hadoop\share\hadoop\yarn\lib\jersey-guice-1.9.jar;C:\hadoop\share\hadoop\yarn\lib\jersey-json-1.9.jar;C:\hadoop\share\hadoop\yarn\lib\jersey-server-1.9.jar;C:\hadoop\share\hadoop\yarn\lib\jettison-1.1.jar;C:\hadoop\share\hadoop\yarn\lib\jetty-6.1.26.jar;C:\hadoop\share\hadoop\yarn\lib\jetty-util-6.1.26.jar;C:\hadoop\share\hadoop\yarn\lib\jline-0.9.94.jar;C:\hadoop\share\hadoop\yarn\lib\jsr305-1.3.9.jar;C:\hadoop\share\hadoop\yarn\lib\leveldbjni-all-1.8.jar;C:\hadoop\share\hadoop\yarn\lib\log4j-1.2.17.jar;C:\hadoop\share\hadoop\yarn\lib\netty-3.6.2.Final.jar;C:\hadoop\share\hadoop\yarn\lib\protobuf-java-2.5.0.jar;C:\hadoop\share\hadoop\yarn\lib\servlet-api-2.5.jar;C:\hadoop\share\hadoop\yarn\lib\stax-api-1.0-2.jar;C:\hadoop\share\hadoop\yarn\lib\xz-1.0.jar;C:\hadoop\share\hadoop\yarn\lib\zookeeper-3.4.6.jar;C:\hadoop\share\hadoop\yarn\hadoop-yarn-api-2.6.5.jar;C:\hadoop\share\hadoop\yarn\hadoop-yarn-applications-distributedshell-2.6.5.jar;C:\hadoop\share\hadoop\yarn\hadoop-yarn-applications-unmanaged-am-launcher-2.6.5.jar;C:\hadoop\share\hadoop\yarn\hadoop-yarn-client-2.6.5.jar;C:\hadoop\share\hadoop\yarn\hadoop-yarn-common-2.6.5.jar;C:\hadoop\share\hadoop\yarn\hadoop-yarn-registry-2.6.5.jar;C:\hadoop\share\hadoop\yarn\hadoop-yarn-server-applicationhistoryservice-2.6.5.jar;C:\hadoop\share\hadoop\yarn\hadoop-yarn-server-common-2.6.5.jar;C:\hadoop\share\hadoop\yarn\hadoop-yarn-server-nodemanager-2.6.5.jar;C:\hadoop\share\hadoop\yarn\hadoop-yarn-server-resourcemanager-2.6.5.jar;C:\hadoop\share\hadoop\yarn\hadoop-yarn-server-tests-2.6.5.jar;C:\hadoop\share\hadoop\yarn\hadoop-yarn-server-web-proxy-2.6.5.jar;C:\hadoop\share\hadoop\mapreduce\lib\aopalliance-1.0.jar;C:\hadoop\share\hadoop\mapreduce\lib\asm-3.2.jar;C:\hadoop\share\hadoop\mapreduce\lib\avro-1.7.4.jar;C:\hadoop\share\hadoop\mapreduce\lib\commons-compress-1.4.1.jar;C:\hadoop\share\hadoop\mapreduce\lib\commons-io-2.4.jar;C:\hadoop\share\hadoop\mapreduce\lib\guice-3.0.jar;C:\hadoop\share\hadoop\mapreduce\lib\guice-servlet-3.0.jar;C:\hadoop\share\hadoop\mapreduce\lib\hadoop-annotations-2.6.5.jar;C:\hadoop\share\hadoop\mapreduce\lib\hamcrest-core-1.3.jar;C:\hadoop\share\hadoop\mapreduce\lib\jackson-core-asl-1.9.13.jar;C:\hadoop\share\hadoop\mapreduce\lib\jackson-mapper-asl-1.9.13.jar;C:\hadoop\share\hadoop\mapreduce\lib\javax.inject-1.jar;C:\hadoop\share\hadoop\mapreduce\lib\jersey-core-1.9.jar;C:\hadoop\share\hadoop\mapreduce\lib\jersey-guice-1.9.jar;C:\hadoop\share\hadoop\mapreduce\lib\jersey-server-1.9.jar;C:\hadoop\share\hadoop\mapreduce\lib\junit-4.11.jar;C:\hadoop\share\hadoop\mapreduce\lib\leveldbjni-all-1.8.jar;C:\hadoop\share\hadoop\mapreduce\lib\log4j-1.2.17.jar;C:\hadoop\share\hadoop\mapreduce\lib\netty-3.6.2.Final.jar;C:\hadoop\share\hadoop\mapreduce\lib\paranamer-2.3.jar;C:\hadoop\share\hadoop\mapreduce\lib\protobuf-java-2.5.0.jar;C:\hadoop\share\hadoop\mapreduce\lib\snappy-java-1.0.4.1.jar;C:\hadoop\share\hadoop\mapreduce\lib\xz-1.0.jar;C:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-app-2.6.5.jar;C:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-common-2.6.5.jar;C:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-core-2.6.5.jar;C:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-hs-2.6.5.jar;C:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-hs-plugins-2.6.5.jar;C:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-jobclient-2.6.5-tests.jar;C:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-jobclient-2.6.5.jar;C:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-shuffle-2.6.5.jar;C:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-examples-2.6.5.jar
STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e8c9fe0b4c252caf2ebf1464220599650f119997; compiled by 'sjlee' on 2016-10-02T23:43Z
STARTUP_MSG: java = 1.8.0_181
************************************************************/
18/07/18 20:44:55 INFO namenode.NameNode: createNameNode [-format]
[Fatal Error] core-site.xml:19:6: The processing instruction target matching "[xX][mM][lL]" is not allowed.
18/07/18 20:44:55 FATAL conf.Configuration: error parsing conf core-site.xml
org.xml.sax.SAXParseException; systemId: file:/C:/hadoop/etc/hadoop/core-site.xml; lineNumber: 19; columnNumber: 6; The processing instruction target matching "[xX][mM][lL]" is not allowed.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2432)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2420)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2491)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1099)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1071)
at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1409)
at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:319)
at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:485)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1375)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1512)
18/07/18 20:44:55 FATAL namenode.NameNode: Failed to start namenode.
java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/C:/hadoop/etc/hadoop/core-site.xml; lineNumber: 19; columnNumber: 6; The processing instruction target matching "[xX][mM][lL]" is not allowed.
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2597)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1099)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1071)
at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1409)
at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:319)
at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:485)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1375)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1512)
Caused by: org.xml.sax.SAXParseException; systemId: file:/C:/hadoop/etc/hadoop/core-site.xml; lineNumber: 19; columnNumber: 6; The processing instruction target matching "[xX][mM][lL]" is not allowed.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2432)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2420)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2491)
... 11 more
18/07/18 20:44:55 INFO util.ExitUtil: Exiting with status 1
18/07/18 20:44:55 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at TheBhaskarDas/192.168.44.1
************************************************************/
C:\hadoop\bin>
core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>C:\hadoop\temp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:50071</value>
</property>
</configuration>
I have fixed it.
I have removed all the characters/anything before <?xml and validated the XML files in https://www.w3schools.com/xml/xml_validator.asp
new core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>\hadoop\temp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:50071</value>
</property>
</configuration>

Error: E0902: Exception occured: [User: Root is not allowed to impersonate root

I am trying to follow the steps given at http://www.rohitmenon.com/index.php/apache-oozie-installation/
Note: I am not using cloudera distibution of hadoop
The above link is similar to http://oozie.apache.org/docs/4.0.1/DG_QuickStart.html
but with more descriptive seems to me
however while running the below command as a root user i am getting exception
./bin/oozie-setup.sh sharelib create -fs
Note: i have two live node shown at dfshealth.jsp . and i have updated the core-site.xml for all three(including namenode) with property as below
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
i understand this is point where i am making mistake Could someone please guide me
Stacktrace
org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [User: root is not allowed to impersonate root]
at
org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:430)
at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:144)
at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:52)
Caused by: org.apache.hadoop.ipc.RemoteException: User: root is not allowed to impersonate root
at org.apache.hadoop.ipc.Client.call(Client.java:1107)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at com.sun.proxy.$Proxy5.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:411)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:135)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:276)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:241)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1411)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1429)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:422)
at org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:420)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:420)
... 2 more
--------------------------------------
Note: Getting E0902: Exception occured: [User: oozie is not allowed to impersonate oozie] i have followed this link as well but not able to solve my problem
if i change the core-site.xml as below only for NameNode
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>[NAMENODE IP]</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>hadoop</value>
</property>
I get the exception as
Unauthorized connection for super-user: hadoop
After adding the property files into core-site.xml restart your hadoop and try. Even though if it not works format the namenode and start hadoop it will work.
You need to add these properties in core-site.xml for impersonation in order to solve your whitelist error
<property>
<name>hadoop.proxyuser.oozie.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.oozie.groups</name>
<value>*</value>
</property>
Hope this fixes your issue.
Follow the advice in the article below. Hadoop before 1.1.0 doesn't support wildcard so you have to explicitly specified the hosts and the groups
http://mail-archives.apache.org/mod_mbox/oozie-user/201212.mbox/%3CCAOcnVr1TZZ5X0Mrb7fFA8JdW6rO6PgoJ9u0=2UYbfXf_o8r=DA#mail.gmail.com%3E
I solved the problem by adding those lines in the core-site.xml-file
hadoop.proxyuser.root.hosts
value = *
hadoop.proxyuser.root.groups
value = *
and it works perfectly all my databases and tables are shown.
./oozie-setup.sh sharelib create -fs hdfs://localhost:9000
try to run this command using sudo.
check for hdfs if this path already exits i.e., /user/user_name/share/lib, if it exists remove it using
hadoop fs -rmr /user/user_name
After that run sudo ./oozied.sh. oozie will be started. Then check for your localhost:11000.

Issue while installing hadoop-2.2.0 in linux 64 bit machine

Using this link ,tried installing Hadoop version - 2.2.0(single node cluster)in ubuntu 12.04(64 bit machine)
http://bigdatahandler.com/hadoop-hdfs/installing-single-node-hadoop-2-2-0-on-ubuntu/
while formatting the hdfs file system via namenode using the following command
hadoop namenode -format
when i'm doing that ,getting the following issue,
14/08/07 10:38:39 FATAL namenode.NameNode: Exception in namenode join
java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/usr/local/hadoop/etc/hadoop/mapred-site.xml; lineNumber: 27; columnNumber: 1; Content is not allowed in trailing section.
What shall i need to do inorder to solve the following issue?
Mapred-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
14/08/07 10:38:39 FATAL namenode.NameNode: Exception in namenode join java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId: file:/usr/local/hadoop/etc/hadoop/mapred-site.xml; lineNumber: 27; columnNumber: 1; Content is not allowed in trailing section
Probably some caracter in your XML that you forget to erase. Please post your full XML. Like #Abhishek said!

Getting the following error "Datanode denied communication with namenode" while configuring hadoop 0.23.8

I am trying to configure hadoop 0.23.8 on my macbook and am running in with the following exception
org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode: 192.168.1.13:50010
at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:549)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:2548)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:784)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:394)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1571)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1567)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1262)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1565)
My core-site.xml looks like this
<configuration>
<property>
<name>dfs.federation.nameservices</name>
<value>ns1</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns1</name>
<value>192.168.1.13:54310</value>
</property>
<property>
<name>dfs.namenode.http-address.ns1</name>
<value>192.168.1.13:50070</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address.ns1</name>
<value>192.168.1.13:50090</value>
</property>
</configuration>
Any ideas on what I may be doing wrong?
Had the same problem with 2.6.0, and shamouda's answer solved it (I was not using dfs.hosts at all so that could not be the answer. I did add
<property>
<name>dfs.namenode.datanode.registration.ip-hostname-check</name>
<value>false</value>
</property>
to hdfs-site.xml and that was enough to fix the issue.
I got the same problem with Hadoop 2.6.0 and the solution for my case was different than Tariq's answer.
I couldn't list the IP-Host mapping in /etc/hosts because I use DHCP for setting the IPs dynamically.
The problem was that my DNS does not allow Reverse DNS lookup (i.e. looking up the hostname given the IP), and HDFS by default use reverse DNS lookup whenever a datanode tries to register with a namenode. Luckily, this behaviour can be disabled by setting this property "dfs.namenode.datanode.registration.ip-hostname-check" to false in hdfs-site.xml
How to know that your DNS does not allow Reverse lookup? the answer in ubuntu is to use the command "host ". If it can resolve the hostname, then reverse lookup is enabled. If it fails, then reverse lookup is disabled.
References:
1. http://rrati.github.io/blog/2014/05/07/apache-hadoop-plus-docker-plus-fedora-running-images/
2. https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
Looks like name resolution issue to me. Possible reasons :
Machine is listed in the file defined by dfs.hosts.exclude
dfs.hosts is used and the machine is not listed within that file
Also make sure you have IP+hostname of the machine listed in your hosts file.
HTH
I got this problem.
earlier configuration in core-site.xml is like this.
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:12345</value>
</property>
Later I've modified the localhost name with my HOSTNAME (PC name)
<property>
<name>fs.default.name</name>
<value>hdfs://cnu:12345</value>
</property>
It worked for me.
Just for information. I have had the same problem and i have recognized, that there was a typo in the hostname of my slaves. Vise versa there the node itself can have the wrong hostname.

Resources