i have Debian 6.03 and problem with best friends hbase and hadoop
step by step, I want working configuration hbase (standalone for the first step) and hadoop :
wget http://www.sai.msu.su/apache//hbase/hbase-0.90.5/hbase-0.90.5.tar.gz
tar xzfv hbase-0.90.5.tar.gz
sudo mv hbase-0.90.5 /usr/local/
sudo ln -s hbase-0.90.5/ hbase
sudo chown -R hduser:hadoop hbase*
lrwxrwxrwx 1 hduser hadoop 13 Янв 21 10:11 hbase -> hbase-0.90.5/
drwxr-xr-x 8 hduser hadoop 4096 Янв 21 10:11 hbase-0.90.5
dan#master:/usr/local/hbase$ su hduser
hduser#master:/usr/local/hbase$ bin/start-hbase.sh
starting master, logging to /usr/local/hbase/bin/../logs/hbase-hduser-master-master.out
hduser#master:/usr/local/hbase$ bin/hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.90.5, r1212209, Fri Dec 9 05:40:36 UTC 2011
hbase(main):001:0> list
TABLE
0 row(s) in 0.8560 seconds
But, after unpack hadoop core v 1.0 in hbase lib/ folder - I got:
hduser#master:/usr/local/hbase$ bin/stop-hbase.sh
hduser#master:/usr/local/hbase$ cp ../hadoop/hadoop-core-1.0.0.jar lib/
hduser#master:/usr/local/hbase$ rm lib/hadoop-core-0.20-append-r1056497.jar
hduser#master:/usr/local/hbase$ bin/start-hbase.sh
starting master, logging to /usr/local/hbase/bin/../logs/hbase-hduser-master-master.out
hduser#master:/usr/local/hbase$ bin/hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.90.5, r1212209, Fri Dec 9 05:40:36 UTC 2011
hbase(main):001:0> list
TABLE
ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
Why I need zookeeper on standalone after replace hadoop-core*.jar?
how to fix it?
Have you configured hbase-env.sh to manage Zookeeper itself?
Have you configured zookeeper quorums in hbase-site.xml?
I have the same problem, and solved it by configuring yarn and map-reduce.
Try this post.
Related
I'm building a small Hadoop cluster composed of 2 nodes : 1 master + 1 worker. I'm using the latest version of Hadoop (3.2) and everything is executed by the root user. In the installation process, I've been able to hdfs namenode -format. Next step is to start the HDFS daemon with start-dfs.sh.
$ start-dfs.sh
Starting namenodes on [master]
bash v3.2+ is required. Sorry.
Starting datanodes
bash v3.2+ is required. Sorry.
Starting secondary namenodes [master]
bash v3.2+ is required. Sorry.
Here's the generated logs in the journal:
$ journalctl --since "1 min ago"
-- Logs begin at Thu 2019-08-29 11:12:27 CEST, end at Thu 2019-08-29 11:46:40 CEST. --
Aug 29 11:46:40 master su[3329]: (to root) root on pts/0
Aug 29 11:46:40 master su[3329]: pam_unix(su-l:session): session opened for user root by root(uid=0)
Aug 29 11:46:40 master su[3329]: pam_unix(su-l:session): session closed for user root
Aug 29 11:46:40 master su[3334]: (to root) root on pts/0
Aug 29 11:46:40 master su[3334]: pam_unix(su-l:session): session opened for user root by root(uid=0)
Aug 29 11:46:40 master su[3334]: pam_unix(su-l:session): session closed for user root
Aug 29 11:46:40 master su[3389]: (to root) root on pts/0
Aug 29 11:46:40 master su[3389]: pam_unix(su-l:session): session opened for user root by root(uid=0)
Aug 29 11:46:40 master su[3389]: pam_unix(su-l:session): session closed for user root
As I'm using Zsh (with Oh-my-Zsh), I logged into a bash console to give it a try. Sadly, I get the same result. In fact, this error happens for all sbin/start-*.sh scripts. However, the hadoop and yarn commands work like a charm.
Since I didn't find much information on this error on the Internet, here I am. Would be glad to have any advice!
Other technical details
Operating system info:
$ lsb_release -d
Description: Debian GNU/Linux 10 (buster)
$ uname -srm
Linux 4.19.0-5-amd64 x86_64
Available Java versions (tried with both):
$ update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 auto mode
* 1 /usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/bin/java 1081 manual mode
2 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode
Some ENV variables you might be interested in:
$ env
USER=root
LOGNAME=root
HOME=/root
PATH=/root/bin:/usr/local/bin:/usr/local/hadoop/bin:/usr/local/hadoop/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SHELL=/usr/bin/zsh
TERM=rxvt-unicode
JAVA_HOME=/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64
HADOOP_HOME=/usr/local/hadoop
HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
ZSH=/root/.oh-my-zsh
Output of the Hadoop executable:
$ hadoop version
Hadoop 3.2.0
Source code repository https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf
Compiled by sunilg on 2019-01-08T06:08Z
Compiled with protoc 2.5.0
From source with checksum d3f0795ed0d9dc378e2c785d3668f39
This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-3.2.0.jar
My Zsh and Bash installation:
$ zsh --version
zsh 5.7.1 (x86_64-debian-linux-gnu)
$ bash --version
GNU bash, version 5.0.3(1)-release (x86_64-pc-linux-gnu)
# only available in a console using *bash*
$ echo ${BASH_VERSINFO[#]}
5 0 3 1 release x86_64-pc-linux-gnu
TL;DR: use a different user (e.g. hadoop) instead of root.
I found the solution but not the deep understanding on what is going on. Despite how sad I can be, here's the solution I found:
Running with root user:
$ start-dfs.sh
Starting namenodes on [master]
bash v3.2+ is required. Sorry.
Starting datanodes
bash v3.2+ is required. Sorry.
Starting secondary namenodes [master_bis]
bash v3.2+ is required. Sorry
Then I created a hadoop user and gave this user privileges on the Hadoop installation (R/W access). After logging in with this new user I have the following output for the command that caused me some troubles:
$ start-dfs.sh
Starting namenodes on [master]
Starting datanodes
Starting secondary namenodes [master_bis]
Moreover, I noticed that processes created by start-yarn.sh were not listed in the output of jps while using Java 11. Switching to Java 8 solved my problem (don't forget to update all $JAVA_HOME variables, both in /etc/environment and hadoop-env.sh).
Success \o/. However, I'd be glad to understand why the root user cannot do this. I know it's a bad habit to use root but in an experimental environment this is not of our interest to have a clean "close-to" production environment. Any information about this will be kindly appreciated :).
try
chsh -s /bin/bash
to change the default shell back to bash
We are setting up automated deployments on a headless system: so using the GUI is not an option here.
Where is start-dfs.sh script for hdfs in Hortonworks Data Platform? CDH / cloudera packages those files under the hadoop/sbin directory. However when we search for those scripts under HDP they are not found:
$ pwd
/usr/hdp/current
Which scripts exist in HDP ?
[stack#s1-639016 current]$ find -L . -name \*.sh
./hadoop-hdfs-client/sbin/refresh-namenodes.sh
./hadoop-hdfs-client/sbin/distribute-exclude.sh
./hadoop-hdfs-datanode/sbin/refresh-namenodes.sh
./hadoop-hdfs-datanode/sbin/distribute-exclude.sh
./hadoop-hdfs-nfs3/sbin/refresh-namenodes.sh
./hadoop-hdfs-nfs3/sbin/distribute-exclude.sh
./hadoop-hdfs-secondarynamenode/sbin/refresh-namenodes.sh
./hadoop-hdfs-secondarynamenode/sbin/distribute-exclude.sh
./hadoop-hdfs-namenode/sbin/refresh-namenodes.sh
./hadoop-hdfs-namenode/sbin/distribute-exclude.sh
./hadoop-hdfs-journalnode/sbin/refresh-namenodes.sh
./hadoop-hdfs-journalnode/sbin/distribute-exclude.sh
./hadoop-hdfs-portmap/sbin/refresh-namenodes.sh
./hadoop-hdfs-portmap/sbin/distribute-exclude.sh
./hadoop-client/sbin/hadoop-daemon.sh
./hadoop-client/sbin/slaves.sh
./hadoop-client/sbin/hadoop-daemons.sh
./hadoop-client/etc/hadoop/hadoop-env.sh
./hadoop-client/etc/hadoop/kms-env.sh
./hadoop-client/etc/hadoop/mapred-env.sh
./hadoop-client/conf/hadoop-env.sh
./hadoop-client/conf/kms-env.sh
./hadoop-client/conf/mapred-env.sh
./hadoop-client/libexec/kms-config.sh
./hadoop-client/libexec/init-hdfs.sh
./hadoop-client/libexec/hadoop-layout.sh
./hadoop-client/libexec/hadoop-config.sh
./hadoop-client/libexec/hdfs-config.sh
./zookeeper-client/conf/zookeeper-env.sh
./zookeeper-client/bin/zkCli.sh
./zookeeper-client/bin/zkCleanup.sh
./zookeeper-client/bin/zkServer-initialize.sh
./zookeeper-client/bin/zkEnv.sh
./zookeeper-client/bin/zkServer.sh
Notice: there are ZERO start/stop sh scripts..
In particular I am interested in the start-dfs.sh script that starts the namenode(s) , journalnode, and datanodes.
How to start DataNode
su - hdfs -c "/usr/lib/hadoop/bin/hadoop-daemon.sh --config /etc/hadoop/conf start datanode";
Github - Hortonworks Start Scripts
Update
Decided to hunt for it myself.
Spun up a single node with Ambari, installed HDP 2.2 (a), HDP 2.3 (b)
sudo find / -name \*.sh | grep start
Found
(a) /usr/hdp/2.2.8.0-3150/hadoop/src/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-dfs.sh
Weird that it doesn't exist in /usr/hdp/current, which should be symlinked.
(b) /hadoop/yarn/local/filecache/10/mapreduce.tar.gz/hadoop/sbin/start-dfs.sh
The recommended way to administer your hadoop cluster would be via the administrator panel. Since you are working on Hotronworks distribution, it makes more sense for you to use Ambari instead.
I installed hadoop 2.6.0 in my laptop running Ubuntu 14.04LTS. I successfully started the hadoop daemons by running start-all.sh and I run a WourdCount example successfully, then I tried to run a jar example that didn't work with me so I decide to format using hadoop namenode -format and start all over again but when I start all daemons using start-dfs.sh && start-yarn.sh then jps all daemons runs but not the datanode as shown bellow:
hdferas#feras-Latitude-E4310:/usr/local/hadoop$ jps
12628 NodeManager
12110 NameNode
12533 ResourceManager
13335 Jps
12376 SecondaryNameNode
How to solve that?
I have faced this issue and it is very easy to solve. Your datanode is not starting because after your namenode and datanode started running you formatted the namenode again. That means you have cleared the metadata from namenode. Now the files which you have stored for running the word count are still in the datanode and datanode has no idea where to send the block reports since you formatted the namenode so it will not start.
Here are the things you need to do to fix it.
Stop all the Hadoop services (stop-all.sh) and close any active ssh connections.
cat /usr/local/hadoop/etc/hadoop/hdfs-site.xml
This step is important, see where datanode's data is gettting stored. It is the value associated for datanode.data.dir. For me it is /usr/local/hadoop/hadoop_data/hdfs/datanode. Open your terminal and navigate to above directory and delete the directory named current which will be there under that directory. Make sure you are only deleting the "current" directory.
sudo rm -r /usr/local/hadoop/hadoop_data/hdfs/datanode/current
Now format the namenode and check whether everything is fine.
hadoop namenode -format
say yes if it asks you for anything.
jps
Hope my answer solves the issue. If it doesn't let me know.
Little advice: Don't format your namenode. Without namenode there is no way to reconstruct the data. If your wordcount is not running that is some other problem.
I had this issue when formatting namenode too. What i did to solve the issue was:
Find your dfs.name.dir location. Consider for example, your dfs.name.dir is /home/hadoop/hdfs.
(a) Now go to, /home/hadoop/hdfs/current.
(b) Search for the file VERSION. Open it using a text editor.
(c) There will be a line namespaceID=122684525 (122684525 is my ID, yours will be different). Note the ID down.
Now find your hadoop.tmp.dir location. Mine is /home/hadoop/temp.
(a) Go to /home/hadoop/temp/dfs/data/current.
(b) Search for the file VERSION and open it using a text editor.
(c) There will be a line namespaceID=. The namespaceID in this file and previous one must be same.
(d) This is the main reason why my datanode was not started. I made them both same and now datanode starts fine.
Note: copy the namespaceID from /home/hadoop/hdfs/current/VERSION to
/home/hadoop/temp/dfs/data/current/VERSION. Dont do it in reverse.
Now do start-dfs.sh && start-yarn.sh. Datanode will be started.
You Just need To Remove All The Contents Of DataNode Folder And Format The Datanode By Using The Following Command
hadoop namenode -format
Even I had same issue and checked the log and found below error
Exception - Datanode log
FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
java.io.IOException: All directories in dfs.datanode.data.dir are invalid: "/usr/local/hadoop_store/hdfs/datanode/
Ran the below command to resolve the issue
sudo chown -R hduser:hadoop /usr/local/hadoop_store
Note - I have create the namenode and datanode under the path /usr/local/hadoop_store
The above problem is occurred due to format the namenode (hadoop namenode -format) without stopping the dfs and yarn daemons. While formating namenode, the question given below is appeared and you press Y key for this.
Re-format filesystem in Storage Directory /tmp/hadoop-root/dfs/name ? (Y or N)
Solution,
You need to delete the files within the current(directory name) directory of dfs.name.dir, you mention in hdfs.site.xml. In my system dfs.name.dir is available in /tmp/hadoop-root/dfs/name/current.
rm -r /tmp/hadoop-root/dfs/name/current
By using the above comment I removed files inside in the current directory. Make sure you are only deleting the "current" directory.Again format the namenode after stopped the dfs and yarn daemons (stop-dfs.sh & stop-yarn.sh). Now datanode will start normally!!
at core-site.xml check for absolute path of temp directory, If this is not pointed correctly or not created (mkdir). The data node cant be started.
add below property in yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
not the right way to do it. but surely works~
remove files from your datanode ,namenode and tmp folder. any files/folders created inside these are owned by hadoop and may have some reference to the last run datanode details which may have failed or locked due to which the datanode does not star at the next attempt
I got the same issue (DataNode & TaskTracker would not come up).
RESOLUTION:
DELETE EVERY "CURRENT" SUB-DIRECTORY UNDER: data, name, and namesecondary to resolve DataNode/taskTracker not showing when you start-all.sh, then jps
(My dfs.name.dir location is: /home/training/hadoop-temp/dfs/data/current; /home/training/hadoop-temp/dfs/name/current; /home/training/hadoop-temp/dfs/namesecondary/current
Make sure you stop services: stop-all.sh
1. Go to each "current" sub-directory under data, name, namesecondary and remove/delete (example: rm -r name/current)
2. Then format: hadoop namenode -format
3. mkdir current under /home/training/hadoop-temp/dfs/data/current
4. Take the directory and contents from /home/training/hadoop-temp/dfs/name/current and copy into the /data/current directory
EXAMPLE: files under:
/home/training/hadoop-temp/dfs/name/current
[training#CentOS current]$ ls -l
-rw-rw-r--. 1 training training 9901 Sep 25 01:50 edits
-rw-rw-r--. 1 training training 582 Sep 25 01:50 fsimage
-rw-rw-r--. 1 training training 8 Sep 25 01:50 fstime
-rw-rw-r--. 1 training training 101 Sep 25 01:50 VERSION
5. Change the storageType=NAME_NODE in VERSION to storageType=DATA_NODE in the data/current/VERSION that you just copied over.
BEFORE:
[training#CentOS dfs]$ cat data/current/VERSION
namespaceID=1018374124
cTime=0
storageType=NAME_NODE
layoutVersion=-32
AFTER:
[training#CentOS dfs]$ cat data/current/VERSION
namespaceID=1018374124
cTime=0
storageType=DATA_NODE
layoutVersion=-32
6. Make sure each subdirectory below has the same files that name/current has for data, name, namesecondary
[training#CentOS dfs]$ pwd
/home/training/hadoop-temp/dfs/
[training#CentOS dfs]$ ls -l
total 12
drwxr-xr-x. 5 training training 4096 Sep 25 01:29 data
drwxrwxr-x. 5 training training 4096 Sep 25 01:19 name
drwxrwxr-x. 5 training training 4096 Sep 25 01:29 namesecondary
7. Now start the services: start-all.sh
You should see all 5 services when you type: jps
I am using hadoop-2.6.0.I resolved using:
1.Deleting all files within
/usr/local/hadoop_store/hdfs
command : sudo rm -r /usr/local/hadoop_store/hdfs/*
2.Format hadoop namenode
command : hadoop namenode -format
3.Go to ..../sbin directory(cd /usr/local/hadoop/sbin)
start-all.sh
use command==> hduser#abc-3551:/$ jps
Following services would be started now :
19088 Jps
18707 ResourceManager
19043 NodeManager
18535 SecondaryNameNode
18329 DataNode
18159 NameNode
When I had this same issue, the 'Current' folder wasn't even being created in my hadoop/data/datanode folder. If this is the case for you too,
~copy the contents of 'Current' from namenode and paste it into datanode folder.
~Then, open VERSION for datanode and change the storageType=NAME_NODE to storageType=DATA_NODE
~run jps to see that the datanode continues to run
I'm running a MapR Community Edition Hadoop cluster (M3).
Unfortunately, the HiveServer2 service crashes and, according the log file in /opt/mapr/hive/hive-0.13/logs/mapr/hive.log, there's a problem with permissions on the scratch directory:
2015-02-24 21:21:08,187 WARN [main]: server.HiveServer2 (HiveServer2.java:init(74)) - Failed to create/change scratchdir permissions to 777: Could not create FileClient java.io.IOException: Could not create FileClient
I checked the settings for the scratch directory using hive -e 'set;' | grep scratch:
hive.exec.scratchdir=/user/mapr/tmp/hive/
hive.scratch.dir.permission=700
I notice that hive.scratch.dir.permission is set to 700 and the error message suggests that it wants to change this to 777. However, according to the filesystem, /mapr/my.cluster.com/user/mapr/tmp has 777 permissions and belongs to the mapr user.
mapr#hadoop01:/mapr/my.cluster.com/user/mapr/tmp$ ls -al
total 2
drwxr-xr-x 3 mapr mapr 1 Feb 22 10:39 .
drwxr-xr-x 5 mapr mapr 3 Feb 24 08:40 ..
drwxrwxrwx 56 mapr mapr 54 Feb 23 10:20 hive
Judging by the filesystem permissions, I would expect the mapr user to do whatever it wants with this folder and so don't understand the error message.
I'm curious to know if anyone's seen this before and, if so, how did you fix it?
Update:
I had a look at the source code, and notice some relevant comments just prior to the warning:
// When impersonation is enabled, we need to have "777" permission on root scratchdir, because
// query specific scratch directories under root scratchdir are created by impersonated user and
// if permissions are not "777" the query fails with permission denied error.
I added set the following properties in hive-site.xml:
<property>
<name>hive.scratch.dir.permission</name>
<value>777</value>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive/</value>
</property>
... and created the /tmp/hive/ folder in HDFS with 777 permissions:
mapr#hadoop01:~$ hadoop fs -ls -d /tmp/hive
drwxrwxrwx - mapr mapr 0 2015-02-27 08:38 /tmp/hive
Although this looked promising, I still got the same warning in hive.log.
Update the permission of your /tmp/hive HDFS directory to set it to 777:
hadoop fs -chmod 777 /tmp/hive
Or remove /tmp/hive , temporary files will be created anyway even when you delete them.
hadoop fs -rm -r /tmp/hive;
rm -rf /tmp/hive
I'm using Amazon EMR (Hadoop2 / AMI version:3.3.1) and I would like to change the default configuration (for example replication factor). In order for the change to take effect I need to restart the cluster and that's where my problems start.
How to do it? The script I found at ./.versions/2.4.0/sbin/stop-dfs.sh doesn't work. The slaves file ./.versions/2.4.0/etc/hadoop/slaves is empty anyway. There are some scripts in init.d:
$ ls -l /etc/init.d/hadoop-*
-rwxr-xr-x 1 root root 477 Nov 8 02:19 /etc/init.d/hadoop-datanode
-rwxr-xr-x 1 root root 788 Nov 8 02:19 /etc/init.d/hadoop-httpfs
-rwxr-xr-x 1 root root 481 Nov 8 02:19 /etc/init.d/hadoop-jobtracker
-rwxr-xr-x 1 root root 477 Nov 8 02:19 /etc/init.d/hadoop-namenode
-rwxr-xr-x 1 root root 1632 Oct 27 21:12 /etc/init.d/hadoop-state-pusher-control
-rwxr-xr-x 1 root root 484 Nov 8 02:19 /etc/init.d/hadoop-tasktracker
but if I for example stop the namenode something will start it again immediately. I looked for documentation and Amazon provides a 600 pages user guide but it's more how to use the cluster and not that much about maintenance.
On EMR 3.x.x , it used traditional SysVInit scripts for managing services. ls /etc/init.d/ can tell you the list of such services. You can restart a service like so,
sudo service hadoop-namenode restart
But if I for example stop the namenode something will start it again
immediately.
However, EMR also has a process called service-nanny that monitors hadoop related services and ensure all of em' are always running. This is the mystery process that brings it back.
So, for truly restarting a service, you would need to stop the service-nanny for a while and then restart/stop the necessary processes. Once you bring back service nanny , it will again do its job. So, you might run commands like -
sudo service service-nanny stop
sudo service hadoop-namenode restart
sudo service service-nanny start
Note that this behavior is different in 4.x.x and 5.x.x AMI's where upstart is used to stop/start applications and service-nanny no longer brings back applications.