I am attempting to install Hadoop 2.8.2 with Java 8 on Mac OS, but i ran into this error after typing the hstart command in terminal. (I initially had Java 9 installed, but switched to Java 8 becuase i thought that Java 9 was causing the issue, but the warning still remains.
WARNING: An illegal reflective access operation has occurred WARNING:
Illegal reflective access by
org.apache.hadoop.security.authentication.util.KerberosUtil
(file:/usr/local/Cellar/hadoop/2.8.2/libexec/share/hadoop/common/lib/hadoop-auth-2.8.2.jar)
to method sun.security.krb5.Config.getInstance() WARNING: Please
consider reporting this to the maintainers of
org.apache.hadoop.security.authentication.util.KerberosUtil WARNING:
Use --illegal-access=warn to enable warnings of further illegal
reflective access operations WARNING: All illegal access operations
will be denied in a future release 18/02/04 23:38:54 WARN
util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable Starting
namenodes on [localhost] localhost: namenode running as process 34039.
Stop it first. localhost: datanode running as process 34125. Stop it
first. Starting secondary namenodes [0.0.0.0] The authenticity of host
'0.0.0.0 (0.0.0.0)' can't be established.
This is in my "/usr/local/Cellar/hadoop/2.8.2/libexec/etc/hadoop/hadoop-env.sh" file:
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc="
Related
When I run spark-submit it works successfully, but the output is not colourised.
(/Users/me/bai/conda-envs/spark-mllib-kmeans) me#my-mbp spark-mllib-kmeans % spark-submit spark-helloWorld.py
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/3.0.1/libexec/jars/spark-unsafe_2.12-3.0.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
20/12/22 12:18:33 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/12/22 12:18:34 INFO SparkContext: Running Spark version 3.0.1
20/12/22 12:18:34 INFO ResourceUtils: ==============================================================
20/12/22 12:18:34 INFO ResourceUtils: Resources for spark.driver:
20/12/22 12:18:34 INFO ResourceUtils: ==============================================================
20/12/22 12:18:34 INFO SparkContext: Submitted application: Simple App
...
I am using Spark version 3.0.1:
(base) me#my-mbp spark-mllib-kmeans % spark-shell --version
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/3.0.1/libexec/jars/spark-unsafe_2.12-3.0.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.0.1
/_/
Using Scala version 2.12.10, OpenJDK 64-Bit Server VM, 14.0.1
Branch HEAD
Compiled by user ubuntu on 2020-08-28T08:58:35Z
Revision 2b147c4cd50da32fe2b4167f97c8142102a0510d
Url https://gitbox.apache.org/repos/asf/spark.git
Type --help for more information.
I am using the default Mac terminal program on latest Mac OS:
% uname -a
Darwin my-mbp.lan 20.2.0 Darwin Kernel Version 20.2.0: Wed Dec 2 20:39:59 PST 2020; root:xnu-7195.60.75~1/RELEASE_X86_64 x86_64
I would like to see the different log statement levels (WARN/INFO/ERROR) in different colours. Perhaps other use of colours to differentiate the output from spark framework and the output from my application.
Given there is so much framework level output and there is noise of WARNINGS due to framework issues, I was hoping better use of colour could help me to scan my output quicker.
Is there a simple solution for this?
I see this behaviour in both native Mac Terminal and MS VSC integrated terminal.
I saw the output line:
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
I see I can create then edit the log4j config file here, so perhaps I just need the config for the log4j config file to colourise all output.
% cd $SPARK_HOME/conf
% cp log4j.properties.template log4j.properties
I got the same warning but it works properly without errors.
I am new in learning Hadoop. And I meet some problems in its configuration. Before it, I finished the configuration of JAVA, SSH, core-site.xml, hdfs-site.xml and hadoop-env.sh. Please tell me how can I solve it. Thank you very much.
root#host1-virtual-machine:/home/host1/usr/hadoop-3.2.1# sbin/start-all.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [host1-virtual-machine]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
2020-02-12 13:19:37,322 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
After I add 5 lines into hadoop-env.sh, it warns the following:
Starting namenodes on [localhost]
localhost: root#localhost: Permission denied (publickey,password).
Starting datanodes
localhost: root#localhost: Permission denied (publickey,password).
Starting secondary namenodes [host1-virtual-machine]
host1-virtual-machine: root#host1-virtual-machine: Permission denied (publickey,password).
2020-02-12 20:26:27,702 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
Starting nodemanagers
localhost: root#localhost: Permission denied (publickey,password).
The reason for this issue is use of different user in installation and for starting the service.
You can define the users as root in hadoop-env.sh as below:
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
Hope this helps
I meet some problems in the configuration of Hadoop3.2.1 for YARN learning. And I found that there are two different conditions in user root and user host1 when I run the sbin/start-all.sh. Can you tell me how to solve it and whether it has connection with the SSH? Thank you very much.
In Root:
root#host1-virtual-machine:/home/host1/usr/hadoop-3.2.1# sbin/start-all.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [host1-virtual-machine]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
2020-02-12 14:40:27,093 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.
In host1 user:
host1#host1-virtual-machine:~/usr/hadoop-3.2.1$ sbin/start-all.sh
WARNING: Attempting to start all Apache Hadoop daemons as host1 in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [localhost]
localhost: host1#localhost: Permission denied (publickey,password).
Starting datanodes
localhost: host1#localhost: Permission denied (publickey,password).
Starting secondary namenodes [host1-virtual-machine]
host1-virtual-machine: host1#host1-virtual-machine: Permission denied (publickey,password).
Starting resourcemanager
Starting nodemanagers
localhost: host1#localhost: Permission denied (publickey,password).
You need to set up passwordless connection between nodes. This link might help
http://mynotesonhadoop.blogspot.com/2017/07/configuring-passwordless-ssh-from.html?m=1
I am getting below error: when i am trying to start hadoop cluster
kishan#kishan-VirtualBox:/usr/local/hadoop/etc/hadoop$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.9.1.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Starting namenodes on [localhost]
localhost: ssh: connect to host localhost port 22: Connection refused
localhost: ssh: connect to host localhost port 22: Connection refused
Starting secondary namenodes [0.0.0.0]
0.0.0.0: ssh: connect to host 0.0.0.0 port 22: Connection refused
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.9.1.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-kishan-resourcemanager-kishan-VirtualBox.out
localhost: ssh: connect to host localhost port 22: Connection refused
I am running into the following issues after using homebrew to install hadoop. I followed the guide here:
http://glebche.appspot.com/static/hadoop-ecosystem/hadoop-hive-tutorial.html
Setting the following environment variables in bashrc:
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home
export HADOOP_INSTALL=/usr/local/Cellar/hadoop/2.3.0
export HADOOP_HOME=$HADOOP_INSTALL
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
After running a hadoop namenode -format.. I attempt to run start-dfs.sh and get the following:
14/05/05 21:19:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: set hadoop variables
localhost: starting namenode, logging to /usr/local/Cellar/hadoop/2.3.0/libexec/logs/mynotebook.local.out
localhost: Error: Could not find or load main class org.apache.hadoop.hdfs.server.namenode.NameNode
localhost: set hadoop variables
localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.3.0/libexec/logs/mynotebook.local.out
localhost: Error: Could not find or load main class org.apache.hadoop.hdfs.server.datanode.DataNode
Starting secondary namenodes [0.0.0.0]
0.0.0.0: set hadoop variables
0.0.0.0: secondarynamenode running as process 12747. Stop it first.
14/05/05 21:19:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
How to I get around this issue?
Based on the first line of the second message,
"14/05/05 21:19:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable"
I suppose that you're running hadoop in a 64 bit operating system. Hadoop is built from default in a 32 bit system, I had the same issue and the same message. What you have to do is re-build hadoop from the source on your system.
I suggest you to use the guide below, it's for the 2.2 version but it's ok for the 2.3 version too
http://csrdu.org/nauman/2014/01/23/geting-started-with-hadoop-2-2-0-building/
Or the official guide
http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/NativeLibraries.html#Build