Run Hadoop Locally on Mac with Java 1.8 - macos

Question

Trying to run single node Hadoop on Mac Sierra.
Here is my configuration:
JAVA Version https://wiki.apache.org/hadoop/HadoopJavaVersions
my-MacBook-Pro:hadoop me$ java -version
java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)
HDFS-SITE
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
CORE-SITE
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
MAPRED-SITE
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9010</value>
</property>
</configuration>
HADOOP-ENV
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home
$ bin/hadoop # displays the usage documentation
Here are some extra settings:
I DID NOT MAKE THIS ACTIVE, WHICH SOME INSTRUCTIONS RECOMMENDED
#export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug"
BY DEFAULT TWO OTHER CONFIGS ALREADY ACTIVE IN HADOOP-ENV
# Some parts of the shell code may do special things dependent upon
# the operating system. We have to set this here. See the next
# section as to why....
export HADOOP_OS_TYPE=${HADOOP_OS_TYPE:-$(uname -s)}
# Under certain conditions, Java on OS X will throw SCDynamicStore errors
# in the system logs.
# See HADOOP-8719 for more information. If one needs Kerberos
# support on OS X, one will want to change/remove this extra bit.
case ${HADOOP_OS_TYPE} in
Darwin*)
export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.realm= "
export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.kdc= "
export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.conf= "
;;
esac
Here is where I formatted namenode:
NAMENODE FORMATTED
My-MacBook-Pro:3.0.0 me$ bin/hdfs namenode -format
WARNING: /usr/local/Cellar/hadoop/3.0.0/libexec/logs does not exist. Creating.
Starting namenode:
2018-04-09 17:45:15,037 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = My-MacBook-Pro.local/10.5.1.16
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 3.0.0
STARTUP_MSG: classpath = /usr/ …
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r c25427ceca461ee979d30edd7a4b0f50718e6533; compiled by 'andrew' on 2017-12-08T19:16Z
STARTUP_MSG: java = 1.8.0_25
************************************************************/
2018-04-09 17:45:15,060 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2018-04-09 17:45:15,076 INFO namenode.NameNode: createNameNode [-format]
2018-04-09 17:45:15,518 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-f925aa84-648a-478b-9134-77d1766a86cb
…
Formatting:
2018-04-09 17:45:17,121 WARN namenode.NameNode: Encountered exception during format:
java.io.IOException: Cannot create directory /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/current
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:416)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:579)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:601)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:167)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1169)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1610)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)
2018-04-09 17:45:17,128 ERROR namenode.NameNode: Failed to start namenode.
Exception:
java.io.IOException: Cannot create directory /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/current
2018-04-09 17:45:17,140 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at My-MacBook-Pro.local/10.5.1.16
************************************************************/
My-MacBook-Pro:3.0.0 me$ sbin/start-dfs.sh
Starting namenodes on [localhost]
Syntax errors:
localhost: /Users/me/.bashrc: line 2: syntax error near unexpected token `('
localhost: /Users/me/.bashrc: line 2: `==> Searching for a previously deleted formula (in the last month)...'
Starting datanodes
localhost: /Users/me/.bashrc: line 2: syntax error near unexpected token `('
localhost: /Users/me/.bashrc: line 2: `==> Searching for a previously deleted formula (in the last month)...'
Starting secondary namenodes [My-MacBook-Pro.local]
My-MacBook-Pro.local: Warning: Permanently added the ECDSA host key for IP address '10.5.13.173' to the list of known hosts.
My-MacBook-Pro.local: /Users/me/.bashrc: line 2: syntax error near unexpected token `('
My-MacBook-Pro.local: /Users/me/.bashrc: line 2: `==> Searching for a previously deleted formula (in the last month)...'
Warning:
2018-04-10 08:28:44,359 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
JPS
My-MacBook-Pro:3.0.0 me$ jps
23201
46872 SecondaryNameNode
46730 DataNode
47039 Jps
Primarily wondering where to begin troubleshooting? That last warning looks problematic. And I do not get anything at http://localhost:9000/, http://localhost:9010/, http://localhost:9870/