i want to run hadoop in my arch linux but i have this error, how i can fix it?
[]# . /usr/lib/hadoop-2.2.0/sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Starting namenodes on [OpenJDK 64-Bit Server VM warning: You have loaded library /usr/lib/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
2013-12-10 23:21:42,602 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable]
Error: Please specify one of --hosts or --hostnames options and not both.
cat: /etc/hadoop/slaves: No such file or directory
Starting secondary namenodes [OpenJDK 64-Bit Server VM warning: You have loaded library /usr/lib/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
2013-12-10 23:21:44,192 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
0.0.0.0]
Error: Please specify one of --hosts or --hostnames options and not both.
starting yarn daemons
starting resourcemanager, logging to /usr/lib/hadoop-2.2.0/logs/yarn-vahid-resourcemanager-kharazi.out
2013-12-10 23:21:47,901 INFO [main] resourcemanager.ResourceManager (StringUtils.java:startupShutdownMessage(601)) - STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting ResourceManager
STARTUP_MSG: host = kharazi/192.168.1.3
STARTUP_MSG: args = []
STARTUP_MSG: version = 2.2.0
STARTUP_MSG: classpath = /etc/hadoop:/etc/hadoop:/etc/hadoop:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/activation-1.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/paranamer-2.3.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/commons-el-1.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/zookeeper-3.4.5.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/xz-1.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/commons-lang-2.5.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/hadoop-annotations-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/hadoop-auth-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/junit-4.8.2.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/commons-net-3.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jettison-1.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/commons-math-2.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/commons-io-2.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/avro-1.7.4.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/guava-11.0.2.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/asm-3.2.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0-tests.jar:/usr/lib/hadoop-2.2.0/share/hadoop/common/hadoop-nfs-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-io-2.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/hdfs/hadoop-hdfs-2.2.0-tests.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/paranamer-2.3.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/hamcrest-core-1.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/guice-3.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/xz-1.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/hadoop-annotations-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/junit-4.10.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/commons-io-2.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/avro-1.7.4.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/asm-3.2.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-site-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-api-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-client-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-common-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-common-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/junit-4.10.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-site-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-api-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-client-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-common-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/hadoop-yarn-server-common-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/paranamer-2.3.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/hamcrest-core-1.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/guice-3.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/xz-1.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/hadoop-annotations-2.2.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/junit-4.10.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/commons-io-2.1.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/avro-1.7.4.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-2.2.0/share/hadoop/yarn/lib/asm-3.2.jar:/etc/hadoop/rm-config/log4j.properties
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common -r 1529768; compiled by 'hortonmu' on 2013-10-07T06:28Z
STARTUP_MSG: java = 1.7.0_45
************************************************************/
cat: /etc/hadoop/slaves: No such file or directory
cat: /etc/hadoop/slaves: No such file or directory
You need to fill in the /etc/hadoop/slaves file with the location of all your slave nodes. You put one slave node hostname per line. Example:
host1
host2
host3
Make sure you can ssh into these nodes without a password.
Related
I'm new to the Hadoop ecosystem,
I installed hadooop 3.3.0 as a Pseudo-Distributed Mode.
The all application http://localhost:8088/ is working but to view name node of the application on http://localhost:9870/ i couldn't (This site can’t be reached).
$ jps
24553 Jps
20537 NodeManager
20429 ResourceManager
and
$ hadoop version
Hadoop 3.3.0
Source code repository https://github.com/apache/hadoop.git -r aa96f1871bfd858f9bac59cf2a81ec470da649af
Compiled by brahma on 2020-07-06T18:21Z
Compiled with protoc 3.7.1
From source with checksum 5dc29b802d6ccd77b262ef9d04d19c4
This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-3.3.0.jar
I tried to restart the process but in vain
$ stop-all.sh
WARNING: Stopping all Apache Hadoop daemons as mhannani in 10 seconds.
WARNING: Use CTRL-C to abort.
Stopping namenodes on [HP]
Stopping datanodes
Stopping secondary namenodes [HP]
2021-01-06 16:42:07,540 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping nodemanagers
Stopping resourcemanager
format :
$ hdfs namenode -format
2021-01-06 16:44:14,683 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = HP/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 3.3.0
and then
$ start-all.sh
WARNING: Attempting to start all Apache Hadoop daemons as mhannani in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [HP]
Starting datanodes
Starting secondary namenodes [HP]
HP: ERROR: Cannot set priority of secondarynamenode process 29847
2021-01-06 16:45:38,266 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
Starting nodemanagers
Please how can i fix that, in order to access my hdfs file system from the browser as it was on earlier version of Hadoop on http://localhost:9870/50075 ?
Any help or advice would be appreciated, Thanks folks.
The issue was not setting correctly the namenode path, and datanode paths of my local file systems :
on $HADOOP_HOME/etc/hadoop/hdfs-site.xml :
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>file://AbsolutePATH/TO/WHERE/THE/namenode/Should/be/stored</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>file://The/same/for/dataNode</value>
</property>
</configuration>
Trying to run single node Hadoop on Mac Sierra.
Here is my configuration:
JAVA Version https://wiki.apache.org/hadoop/HadoopJavaVersions
my-MacBook-Pro:hadoop me$ java -version
java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)
HDFS-SITE
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
CORE-SITE
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
MAPRED-SITE
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9010</value>
</property>
</configuration>
HADOOP-ENV
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home
$ bin/hadoop # displays the usage documentation
Here are some extra settings:
I DID NOT MAKE THIS ACTIVE, WHICH SOME INSTRUCTIONS RECOMMENDED
#export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug"
BY DEFAULT TWO OTHER CONFIGS ALREADY ACTIVE IN HADOOP-ENV
# Some parts of the shell code may do special things dependent upon
# the operating system. We have to set this here. See the next
# section as to why....
export HADOOP_OS_TYPE=${HADOOP_OS_TYPE:-$(uname -s)}
# Under certain conditions, Java on OS X will throw SCDynamicStore errors
# in the system logs.
# See HADOOP-8719 for more information. If one needs Kerberos
# support on OS X, one will want to change/remove this extra bit.
case ${HADOOP_OS_TYPE} in
Darwin*)
export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.realm= "
export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.kdc= "
export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.conf= "
;;
esac
Here is where I formatted namenode:
NAMENODE FORMATTED
My-MacBook-Pro:3.0.0 me$ bin/hdfs namenode -format
WARNING: /usr/local/Cellar/hadoop/3.0.0/libexec/logs does not exist. Creating.
Starting namenode:
2018-04-09 17:45:15,037 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = My-MacBook-Pro.local/10.5.1.16
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 3.0.0
STARTUP_MSG: classpath = /usr/ …
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r c25427ceca461ee979d30edd7a4b0f50718e6533; compiled by 'andrew' on 2017-12-08T19:16Z
STARTUP_MSG: java = 1.8.0_25
************************************************************/
2018-04-09 17:45:15,060 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2018-04-09 17:45:15,076 INFO namenode.NameNode: createNameNode [-format]
2018-04-09 17:45:15,518 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-f925aa84-648a-478b-9134-77d1766a86cb
…
Formatting:
2018-04-09 17:45:17,121 WARN namenode.NameNode: Encountered exception during format:
java.io.IOException: Cannot create directory /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/current
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:416)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:579)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:601)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:167)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1610)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)
2018-04-09 17:45:17,128 ERROR namenode.NameNode: Failed to start namenode.
Exception:
java.io.IOException: Cannot create directory /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/current
2018-04-09 17:45:17,140 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at My-MacBook-Pro.local/10.5.1.16
************************************************************/
My-MacBook-Pro:3.0.0 me$ sbin/start-dfs.sh
Starting namenodes on [localhost]
Syntax errors:
localhost: /Users/me/.bashrc: line 2: syntax error near unexpected token `('
localhost: /Users/me/.bashrc: line 2: `==> Searching for a previously deleted formula (in the last month)...'
Starting datanodes
localhost: /Users/me/.bashrc: line 2: syntax error near unexpected token `('
localhost: /Users/me/.bashrc: line 2: `==> Searching for a previously deleted formula (in the last month)...'
Starting secondary namenodes [My-MacBook-Pro.local]
My-MacBook-Pro.local: Warning: Permanently added the ECDSA host key for IP address '10.5.13.173' to the list of known hosts.
My-MacBook-Pro.local: /Users/me/.bashrc: line 2: syntax error near unexpected token `('
My-MacBook-Pro.local: /Users/me/.bashrc: line 2: `==> Searching for a previously deleted formula (in the last month)...'
Warning:
2018-04-10 08:28:44,359 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
JPS
My-MacBook-Pro:3.0.0 me$ jps
23201
46872 SecondaryNameNode
46730 DataNode
47039 Jps
Primarily wondering where to begin troubleshooting? That last warning looks problematic. And I do not get anything at http://localhost:9000/, http://localhost:9010/, http://localhost:9870/
I installed Hadoop and Spark via Homebrew
$ brew list --versions | grep spark
apache-spark 2.2.0
$ brew list --versions | grep hadoop
hadoop 2.8.1 2.8.2 hdfs
where Hadoop 2.8.2 is what I am using.
I followed this post to configure Hadoop. Also, followed this post to configure spark.yarn.archive as:
spark.yarn.archive hdfs://localhost:9000/user/panc25/spark-jars.zip
The following are my Hadoop/Spark related environment setting in my .bash_profile :
# ---------------------
# Hadoop
# ---------------------
export HADOOP_HOME=/usr/local/Cellar/hadoop/2.8.2
export YARN_CONF_DIR=$HADOOP_HOME/libexec/etc/hadoop/
alias hadoop-start="$HADOOP_HOME/sbin/start-dfs.sh;$HADOOP_HOME/sbin/start-yarn.sh"
alias hadoop-stop="$HADOOP_HOME/sbin/stop-yarn.sh;$HADOOP_HOME/sbin/stop-dfs.sh"
# ---------------------
# Apache Spark
# ---------------------
export SPARK_HOME=/usr/local/Cellar/apache-spark/2.2.0/libexec
export PATH=$SPARK_HOME/../bin:$SPARK_HOME/sbin:$PATH
I can successfully start hadoop (hdfa + yarn):
$ hadoop-start
17/11/12 17:08:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/hadoop-panc25-namenode-mbp13mid2017.local.out
localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/hadoop-panc25-datanode-mbp13mid2017.local.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/hadoop-panc25-secondarynamenode-mbp13mid2017.local.out
17/11/12 17:08:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/yarn-panc25-resourcemanager-mbp13mid2017.local.out
localhost: starting nodemanager, logging to /usr/local/Cellar/hadoop/2.8.2/libexec/logs/yarn-panc25-nodemanager-mbp13mid2017.local.out
$ jps
92723 NameNode
93188 Jps
93051 ResourceManager
93149 NodeManager
92814 DataNode
92926 SecondaryNameNode
However, when I start spark-shell --master yarn it seems to freeze and I don't know what is going on:
What is wrong?
BTW, I could visit the SparkUI http://localhost:4040/, but all pages are blank.
I experienced a similar issue an was caused by the fact that I forgot to append /conf to HADOOP_CONF_DIR env variable (/etc/hadoop/conf).
In my case I was running spark 2.1 cloudera distribution and specified HADOOP_CONF_DIR=/etc/hadoop/conf/:/etc/hive/conf/ . Due to some reason it was getting stuck so I modified it to HADOOP_CONF_DIR=/etc/hadoop/conf/ and it worked. Still looking for the root cause !
I'm using mac and java version:
$java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
followed this link: https://dtflaneur.wordpress.com/2015/10/02/installing-hadoop-on-mac-osx-el-capitan/
I first brew install hadoop, config ssh connection and xml files as required, and
start-dfs.sh
start-yarn.sh
The screen output is like this:
$start-dfs.sh
17/05/06 09:58:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: namenode running as process 74213. Stop it first.
localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.7.3/libexec/logs/hadoop-x-datanode-xdeMacBook-Pro.local.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 74417. Stop it first.
17/05/06 09:58:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
$start-dfs.sh
17/05/06 09:58:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: namenode running as process 74213. Stop it first.
localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.7.3/libexec/logs/hadoop-x-datanode-xdeMacBook-Pro.local.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 74417. Stop it first.
17/05/06 09:58:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Then using jps I cannot see "DataNode" and "ResourceManager". I suppose DataNode is hdfs module and ResourceManager is yarn module:
$jps
74417 SecondaryNameNode
75120 Jps
74213 NameNode
74539 ResourceManager
74637 NodeManager
I can list hdfs files:
$hdfs dfs -ls /
17/05/06 09:58:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x - x supergroup 0 2017-05-05 23:50 /user
But running the pi examples throws exception:
$hadoop jar /usr/local/Cellar/hadoop/2.7.3/libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar pi 2 5
Number of Maps = 2
Samples per Map = 5
17/05/06 10:19:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/06 10:19:49 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/x/QuasiMonteCarlo_1494037188550_135794067/in/part0 could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
I wonder if I missed any configuation, how can I make sure that they run successfully, and how to check or trouble shoot possible failure reasons?
Thanks.
I am too in learning phase yet. This error comes when there is no datanode available to read/write.
You can check Resource Manager using this URL: http://localhost:50070
Is there any datanode running or not.
For trouble shooting you can check logs generated under installation directory of hadoop . If you can share that logs i can try to help.
I had Hadoop 2.4 this morning (see my previous 2 questions). Now I removed it and installed 2.2 as I had issues with 2.4, and also as I think 2.2 is the latest stable release. Now I followed the tutorial here:
http://codesfusion.blogspot.com/2013/10/setup-hadoop-2x-220-on-ubuntu.html?m=1
I am pretty sure I did everything right but I am facing similar issues again.
When I run jps it is obvious that the data node is not starting up.
What am I doing wrong again?
hduser#test02:~$ start-dfs.sh
14/06/06 18:12:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Starting namenodes on []
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hduser-namenode-test02.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hduser-datanode-test02.out
localhost: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
localhost: It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hduser-secondarynamenode-test02.out
0.0.0.0: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
0.0.0.0: It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
14/06/06 18:13:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hduser#test02:~$ jps
2201 Jps
hduser#test02:~$ jps
2213 Jps
hduser#test02:~$ start-yarn
start-yarn: command not found
hduser#test02:~$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-test02.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-test02.out
hduser#test02:~$ jps
2498 NodeManager
2264 ResourceManager
2766 Jps
hduser#test02:~$ jps
2784 Jps
2498 NodeManager
2264 ResourceManager
hduser#test02:~$ jps
2498 NodeManager
2264 ResourceManager
2796 Jps
hduser#test02:~$
My problem was that I took these instructions from the tutorial too literally.
Paste following between <configuration>
fs.default.name
hdfs://localhost:9000
I suspected this was wrong while doing it but still I did it.
It seemed incorrect as the core-site.xml file is in XML format.
So actually, it needs to look like this.
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
Changing it to this fixed my problem.
I had similar issues with DataNode not starting up. What I did was reformat the namenode, then restarted the cluster. Then, running jps confirmed that data node was started up.
This can be caused by placing the HDFS directory in your "home" directory (on a linux box) since upon starting up and shutting down the OS affects these folders (not exactly sure how, but to prevent this problem in the future, move the HDFS directory out of your home directory).
Please let me know if this works.