I have configured Hadoop 2.6.0 successfully. Next, I am trying to install Hbase 0.98.9 but am having trouble starting up Hbase.
I get the below error message:
Error: Could not find or load main class org.apache.hadoop.hbase.util.HBaseConfTool
Error: Could not find or load main class org.apache.hadoop.hbase.zookeeper.ZKServerTool
starting master, logging to /usr/local/hbase/logs/hbase-yarn-master-hadoopmaster.out
Error: Could not find or load main class org.apache.hadoop.hbase.master.HMaster
localhost:
starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-yarn-regionserver-hadoopmaster.out
localhost: Error: Could not find or load main class org.apache.hadoop.hbase.regionserver.HRegionServer
And, this is my hbase-site.xml file
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://hadoopmaster:9000/hbase</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/yarn/hbase/zookeeper</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
</configuration>
Please let me know what is wrong with my configuration.
Regards.
Add this line in hadoop-env.sh:
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/hbase/jars
NOTE: Change /path/to/hbase/jars to hbase jars location. If possible add all available hbase jar files to hadoop classpath (to avoid future class problems).
Related
I want to install Java and Hadoop on Windows10. So far I have Java jdk1.8.0_211 as when I try
c:\>java -version
it returns:
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)
and when I try
c:\>javac -version
it returns:
javac 1.8.0_211
During hadoop-2.8.0 installation I try:
C:\hadoop\hadoop-2.8.0>hdfs namenode -format
which returns:
Error: Could not find or load main class
The same appears when I try:
c:\>hadoop --version
Again it returns:
Error: Could not find or load main class
When I try:
C:\hadoop\hadoop-2.8.0\sbin>start-all.cmd
it returns:
This script is Deprecated. Instead use start-dfs.cmd and start-yarn.cmd
starting yarn daemons
How can I fix this main class problem practically?
Thank you for your time and effort.
What I already did:
I have no idea what the main class is. I googled and read almost everywhere. I found a lot of theoretical stuff about main class and classpath etc which gives an idea but essentially is hard to understand for me. As I am only a hadoop user, no data engineer nor Java programmer, I appreciate a practical solution.
I only try to follow installation guides. As there are many guides which does not exactly what I want (maybe commercial interests?!) , I have set up my own installation plan:
(have I taken the right steps?)
Hadoop installation plan
Download
Download Java
jdk-8u211-windows-x64.exe (jdk 64x version!)
from Oracle
https://www.oracle.com/technetwork/pt/java/javase/downloads/jdk8-downloads-2133151.html?printOnly=1
Download
Hadoop 2.8.0 (binary version!)
hadoop-2.8.0.tar.gz
from Apache Hadoop
https://hadoop.apache.org/releases.html
Install
double click
jdk-8u211-windows-x64.exe
Extract
With WinRAR as admin
hadoop-2.8.0.tar.gz
To Locations
C:\Java\ (for jdk1.8.0_211)
>>> C:\Java\jdk1.8.0_211
C:\Java\jdk1.8.0_211\ (for jre1.8.0_211)
>>> C:\Java\jdk1.8.0_211\jre1.8.0_211
C:\hadoop\ (for hadoop-2.8.0)
>>> C:\hadoop\hadoop-2.8.0
Add folders
C:\hadoop\hadoop-2.8.0\data\datanode
C:\hadoop\hadoop-2.8.0\data\namenode
User variables for admin
1 New and add
JAVA_HOME= C:\Java\jdk1.8.0_211
HADOOP_HOME=”C:\hadoop\hadoop-2.8.0\
2 Edit path, add
C:\Java\jdk1.8.0_211\bin
C:\hadoop\hadoop-2.8.0\bin
C:\hadoop\hadoop-2.8.0\sbin
C:\hadoop\hadoop-2.8.0\share\hadoop\common\*
C:\hadoop\hadoop-2.8.0\share\hadoop\hdfs
C:\hadoop\hadoop-2.8.0\share\hadoop\hdfs\lib\*
C:\hadoop\hadoop-2.8.0\share\hadoop\hdfs\*
C:\hadoop\hadoop-2.8.0\share\hadoop\yarn\lib\*
C:\hadoop\hadoop-2.8.0\share\hadoop\yarn\*
C:\hadoop\hadoop-2.8.0\share\hadoop\mapreduce\lib\*
C:\hadoop\hadoop-2.8.0\share\hadoop\mapreduce\*
C:\hadoop\hadoop-2.8.0\share\hadoop\common\lib\*
System variables
1 New and add
JAVA_HOME=C:\Java\jdk1.8.0_211
HADOOP_HOME=C:\hadoop\hadoop-2.8.0
2 Edit path,
New,
Add
%JAVA_HOME%
%JAVA_HOME%\bin
%HADOOP_HOME%
%HADOOP_HOME%\bin
%HADOOP_HOME%\sbin
%HADOOP_HOME%\etc\hadoop,
%HADOOP_HOME%\share\hadoop\common\* ,
%HADOOP_HOME%\share\hadoop\common\lib\* ,
%HADOOP_HOME%\share\hadoop\hdfs\* ,
%HADOOP_HOME%\share\hadoop\hdfs\lib\* ,
%HADOOP_HOME%\share\hadoop\mapreduce\* ,
%HADOOP_HOME%\share\hadoop\mapreduce\lib\* ,
%HADOOP_HOME%\share\hadoop\yarn\* ,
%HADOOP_HOME%\share\hadoop\yarn\lib\ *
Make mapred-site.xml file
Copy
C:\hadoop\hadoop-2.8.0\etc\hadoop\mapred-site.xml.template
rename to
C:\hadoop\hadoop-2.8.0\etc\hadoop\mapred-site.xml
Configure files by entering code between
<configuration> and </configuration>
in all of those 4 files:
C:\hadoop\hadoop-2.8.0\etc\hadoop\core-site.xml
C:\hadoop\hadoop-2.8.0\etc\hadoop\hdfs-site.xml
C:\hadoop\hadoop-2.8.0\etc\hadoop\mapred-site.xml
C:\hadoop\hadoop-2.8.0\etc\hadoop\yarn-site.xml
Code to enter:
1 Code for core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
Alternatively:
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
2 Code for hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>C:\hadoop\hadoop-2.8.0\data\namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>C:\hadoop\hadoop-2.8.0\data\datanode</value>
</property>
3 Code for mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
4 Code for yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
Config hadoop-env.cmd:
C:\hadoop\hadoop-2.8.0\etc\hadoop\hadoop-env.cmd
change
set JAVA_HOME=%JAVA_HOME%
to:
#rem set JAVA_HOME=%JAVA_HOME%
set JAVA_HOME= C:\Java\jdk1.8.0_211
hdfs namenode –format Cmd line:
cd C:\hadoop\hadoop-2.8.0
hdfs namenode –format
returns:
Error: Could not find or load main class
testing in Cmd line:
java -version
javac -version
echo %JAVA_HOME%
echo %HADOOP_HOME%
are all ok, but
hadoop --version
returns:
Error: Could not find or load main class
starting hadoop in cmd line:
cd C:\hadoop\hadoop-2.8.0\sbin
start-all.cmd
it returns:
This script is Deprecated. Instead use start-dfs.cmd and start-yarn.cmd
starting yarn daemons
start-dfs.cmd
it returns:
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
start-yarn.cmd
it returns:
starting yarn daemons
Browser
http://localhost:8088 appears
http://localhost:50070 not OK
stop hadoop in cmd line:
cd C:\hadoop\hadoop-2.8.0\sbin
stop-all.cmd
stop-dfs.cmd
stop-yarn.cmd
I'm running spark on yarn.
My spark versoin is 2.1.1, and hadoop version is apache hadoop 2.7.3.
when a spark job running on yarn in cluster mode, I can view the Executor's log via the stdout/stderr links like
http://hadoop-slave1:8042/node/containerlogs/container_1500432603585_0148_01_000001/hadoop/stderr?start=-4096
but when the job completed, view the Executor's log via the stdout/stderr links will get an error page like
Redirecting to log server for container_1500432603585_0148_01_000001
java.lang.Exception: Unknown container. Container either has not
started or has already completed or doesn't belong to this node at
all.
And then it will auto redirect to
http://hadoop-slave1:8042/node/hadoop-master:19888/jobhistory/logs/hadoop-slave1:36207/container_1500432603585_0148_01_000001/container_1500432603585_0148_01_000001/hadoop
and get other error page like
Sorry, got error 404
Please consult RFC 2616 for meanings of the error code.
Error Details
org.apache.hadoop.yarn.webapp.WebAppException: /hadoop-master:19888/jobhistory/logs/hadoop-slave1:50284/container_1500432603585_0145_01_000002/container_1500432603585_0145_01_000002/oryx: controller for hadoop-master:19888 not found
at org.apache.hadoop.yarn.webapp.Router.resolveDefault(Router.java:232)
at org.apache.hadoop.yarn.webapp.Router.resolve(Router.java:140)
at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:134)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
Actually i can visit the Executor's log using this url when the
spark job completed:
http://hadoop-master:19888/jobhistory/logs/hadoop-slave1:36207/container_1500432603585_0148_01_000001/container_1500432603585_0148_01_000001/hadoop
it's a little different from the previous url, it remove the head "hadoop-slave1:8042/node/".
Does anyone knows another better method to view the spark logs when the spark job completed ?
I have configed the yarn-site.xml
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop-master</value>
<description>The hostname of the RM.</description>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>${yarn.resourcemanager.hostname}:19888/jobhistory/logs</value>
</property>
and mapred-site.xml
<property>
<name>mapreduce.jobhistory.address</name>
<value>${yarn.resourcemanager.hostname}:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.admin.address </name>
<value>${yarn.resourcemanager.hostname}:10033</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>${yarn.resourcemanager.hostname}:19888</value>
</property>
I have encounter this situation.view the completed spark steaming job logs through YARN UI History tab, but get error below:
Failed while trying to construct the redirect url to the log server. Log Server url may not be configured
java.lang.Exception: Unknown container. Container either has not started or has already completed or doesn't belong to this node at all.
The solution is configure the file yarn-site.xml. Add key yarn.log.server.url :
<property>
<name>yarn.log.server.url</name>
<value>http://<LOG_SERVER_HOSTNAME>:19888/jobhistory/logs</value>
</property>
Then restart yarn cluster to reload yarn-site.xml.(this step is important!)
I have just installed Hbase through brew install hbase. Edited hbase-site.xml
<configuration>
<property>
<name>hbase.rootdir</name>
<value>file:///usr/local/Cellar/hbase/databases/hbase-${user.name}/hbase</value>
<description>The directory shared by region servers and into
which HBase persists. The URL should be 'fully-qualified'
to include the filesystem scheme. For example, to specify the
HDFS directory '/hbase' where the HDFS instance's namenode is
running at namenode.example.org on port 9000, set this value to:
hdfs://namenode.example.org:9000/hbase. By default HBase writes
into /tmp. Change this configuration else all data will be lost
on machine restart.
</description>
</property>
</configuration>
Exported JAVA_HOME and HBASE_HOME.
When i'm trying to start i m getting following exception:
Abhisheks-MacBook-Pro:bin abhishek$ start-hbase.sh
Error: Could not find or load main class org.apache.hadoop.hbase.util.HBaseConfTool
Error: Could not find or load main class org.apache.hadoop.hbase.zookeeper.ZKServerTool
starting master, logging to /usr/local/Cellar/hbase/1.0.0/logs/hbase-abhishek-master-Abhisheks-MacBook-Pro.local.out
Error: Could not find or load main class org.apache.hadoop.hbase.master.HMaster
cat: /usr/local/Cellar/hbase/1.0.0/conf/regionservers: No such file or directory
cat: /usr/local/Cellar/hbase/1.0.0/conf/regionservers: No such file or directory
I have Hadoop2.6.0 and Hbase1.0.0. Though i'm seeing many people have already faced this problem but i cannot find the solution. What else needs to be done to start Hbase without any issue?
Solution:
HBASE_HOME=/usr/local/Cellar/hbase/1.0.0/libexec
it should be configured such that conf folder lies in HBASE_HOME directory.
Checking master-status:
localhost:60010
edit hbase-site.xml
<configuration>
<property>
<name>hbase.rootdir</name>
<value>file:///usr/local/Cellar/hbase/databases/hbase-${user.name}/hbase</value>
<description>The directory shared by region servers and into
which HBase persists. The URL should be 'fully-qualified'
to include the filesystem scheme. For example, to specify the
HDFS directory '/hbase' where the HDFS instance's namenode is
running at namenode.example.org on port 9000, set this value to:
hdfs://namenode.example.org:9000/hbase. By default HBase writes
into /tmp. Change this configuration else all data will be lost
on machine restart.
</description>
</property>
<property >
<name>hbase.master.port</name>
<value>60000</value>
<description>The port the HBase Master should bind to.</description>
</property>
<property>
<name>hbase.master.info.port</name>
<value>60010</value>
<description>The port for the HBase Master web UI.
Set to -1 if you do not want a UI instance run.</description>
</property>
</configuration>
I am trying to install hbase-0.96.0-hadoop2 on Hadoop 2.2.0. While I am trying to start my HBase. HBase is giving following error.
master: log4j:ERROR Could not find value for key log4j.appender.DRFAS
master: log4j:ERROR Could not instantiate appender named "DRFAS".
log4j:ERROR Could not find value for key log4j.appender.DRFAS
log4j:ERROR Could not instantiate appender named "DRFAS".
When I am doing JPS Linux is showing following processes:
17422 JobHistoryServer
11461 NameNode
31375 Jps
12127 ResourceManager
11671 DataNode
30077 HRegionServer
12344 NodeManager
11935 SecondaryNameNode
30948 HQuorumPeer
Here is my hbase-site.xml configuraiton:
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://master:9000/hbase</value>
<description>The directory shared by RegionServers.
</description>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
</description>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>master</value>
</property>
<property>
<name>zookeeper.znode.parent</name>
<value>/master</value>
</property>
</configuration>
Try these two methods .
Stop your hbase demon and clear the hbase log files which was located
in /tmp/ folder delete all files which had name hbase in it
after deleting disconnect your machine from internet and try to
start the hbase demon now.
Hbase has this weird issue in some x64 ubuntu machines disconnecting from internet will help in resolving this issue,after startup you can connect to the internet.
now try to access hbase from cli
bin/hbase
I deployed Hadoop 0.23.6 in Ubuntu 12.04 LTS. I am able to copy files across and do file manipulation. I am using YARN for mapreduce.
I am getting the following error, when I am trying to run any mapreduce application using the hadoop-mapreduce-examples-0.23.6.jar
Command used:
bin/hadoop jar hadoop-mapreduce-examples-0.23.6.jar randomwriter -Dmapreduce.randomwriter.mapsperhost=1 -Dmapreduce.job.user.name=$USER -Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912 -Ddfs.block.size=536870912 -libjars hadoop-mapreduce-client-app-0.23.6.jar output
Hadoop version: 0.23.6
Container launch failed for container_1364342550899_0001_01_000002 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1364342550899_0001_m_000000_0
Verify your yarn-site.xml configuration. You need to have below properties configured.
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
For more details, have look at jira
https://issues.apache.org/jira/browse/MAPREDUCE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel