Copy data from local disk to HDFS in a federation of HDFS - hadoop

I use two hosts (hadoop-coc-1 and hadoop-coc-2) to try to configure a Federation of HDFS in them. Therefore, I configured the $ cat etc/hadoop/hdfs-site.xml in both hosts (hadoop-coc-1, and hadoop-coc-2).
Both HDFS are running properly with the WebHDFS. But, with this configuration, the defaultFS service is not running. Maybe this is a reason that I can't copy data from local disk to HDFS.
I have tried to copy data with the following commands:
$ hdfs dfs -copyFromLocal ~/input1 webhdfs://hadoop-coc-1/
Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
15/03/02 12:32:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
copyFromLocal: Connection refused
$ curl -i -X PUT -T ~/input1 "http://hadoop-coc-1:8080/?op=CREATE"
HTTP/1.1 100 Continue
HTTP/1.1 405 HTTP method PUT is not supported by this URL
Date: Mon, 02 Mar 2015 17:34:35 GMT
Pragma: no-cache
Date: Mon, 02 Mar 2015 17:34:35 GMT
Pragma: no-cache
Content-Length: 0
Server: Jetty(6.1.26)
I don't understand why I can't copy data from local disk to HDFS that is using WebHDfS protocol. Any help to understand this error? Why I can't copy data from the local disk to HDFS? Did I configure well the hdfs-site.xml?
hdfs-site.xml in hadoop-coc-1 and hadoop-coc-2 hosts.
<configuration>
<property>
<name>dfs.nameservices</name>
<value>ns1,ns2</value>
</property>
<property>
<name>fs.defaultFS.ns1</name>
<value>hdfs://hadoop-coc-1:9000</value>
</property>
<property>
<name>fs.defaultFS.ns2</name>
<value>hdfs://hadoop-coc-2:9000</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns1</name>
<value>hadoop-coc-1:6600</value>
</property>
<property>
<name>dfs.namenode.http-address.ns1</name>
<value>hadoop-coc-1:8080</value>
</property>
<property>
<name>dfs.namenode.secondaryhttp-address.ns1</name>
<value>hadoop-coc-1:8080</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns2</name>
<value>hadoop-coc-2:6600</value>
</property>
<property>
<name>dfs.namenode.http-address.ns2</name>
<value>hadoop-coc-2:8080</value>
</property>
<property>
<name>dfs.namenode.secondaryhttp-address.ns2</name>
<value>hadoop-coc-2:8080</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.support.append</name>
<value>true</value>
</property>
</configuration>
Federation of HDFS allows to have several namespaces to access HDFS. With my configuration (see hdfs-site.xml), hadoop-coc-1 and hadoop-coc-2 host have a namespace to the same HDFS. I wonder if it is possible to have two YARN runtimes running in each host that uses the same HDFS?

The default web server port is 50070 and the data is streamed from datanodes by webservers on port 50075. Had you changed this configuration somewhere?
Furthermore, when you try to copy data to the webdfs you specify no port
http://hadoop.apache.org/docs/r1.0.4/webhdfs.html

Related

Hadoop: Secondary NameNode Permission Denied

I'm attempting to run Hadoop in pseudo-distributed mode to learn how the system work. To install it, I've downloaded Hadoop-3.0.0 from the site, untarred it. I've done my configurations as follows (leaving out the configuration tags for brevity):
core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost/</value>
</property>
hdsf-site.xml
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
yarn-site.xml
<property>
<name>yarn.resourcemanager.hostname</name>
<value>localhost</value> </property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
After doing this, I've formatted my hdfs using
hdfs namenode -format
I've also setup passwordless ssh using the following:
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa2
cat ~/.ssh/id_rsa2.pub >> ~/.ssh/authorized_keys
(I've also added id_rsa2.pub as the default for localhost using a config file, since I already was using id_rsa.pub for something else and didn't want to mix-and-match in case I broke something)
I'm able to ssh into localhost. All looks well.
Then I run start-dfs.sh, and I see this error:
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [zm.local]
zm.local: zm#zm.local: Permission denied (publickey,password,keyboard-interactive).
2018-01-16 17:31:35,807 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
If I run jps (after starting yarn and mapreduce history server), I have the following:
37921 NodeManager
38070 Jps
37434 NameNode
38060 JobHistoryServer
37821 ResourceManager
Noticeably, the SecondaryNameNode is missing, my assumption being it's due to the error above.
I can then try to use hadoop's fs command and I'm able to create a folder and look it up. But if I try to copy any data over, I get notified that the NameNode is in SAFEmode. If I turn off save mode using:
hdfs dfsadmin -safemode leave
It immediately turns back on. By going to the namenode port on localhost, I see the following message:
Safe mode is ON. Resources are low on NN. Please add or free up more resourcesthen turn off safe mode manually. NOTE: If you turn off safe mode before adding resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off.
However, I have plenty of resources. The single datanode is using less than 8% of it's allotted space, the namenode as almost 100GB of space. The datanode and namenode are both reporting as healthy. Thus, I think the problem is the lack of a secondary namenode. With that in mind, is anyone aware what might be causing the SecondaryNameNode to have different permission issues from the PrimaryNameNode? It seems to be trying to put the sNN somewhere on the local machine instead - but when I check in /tmp/hadoop*, all of the file permissions seem to be normal.
Thanks for any help.

httpfs error Operation category READ is not supported in state standby

I am working on hadoop apache 2.7.1 and I have a cluster that consists of 3 nodes
nn1
nn2
dn1
nn1 is the dfs.default.name, so it is the master name node.
I have installed httpfs and started it of course after restarting all the services. When nn1 is active and nn2 is standby I can send this request
http://nn1:14000/webhdfs/v1/aloosh/oula.txt?op=open&user.name=root
from my browser and a dialog of open or save for this file appears, but when I kill the name node running on nn1 and start it again as normal then because of high availability nn1 becomes standby and nn2 becomes active.
So here httpfs should work, even if nn1 becomes stand by, but sending the same request now
http://nn1:14000/webhdfs/v1/aloosh/oula.txt?op=open&user.name=root
gives me the error
{"RemoteException":{"message":"Operation category READ is not supported in state standby","exception":"RemoteException","javaClassName":"org.apache.hadoop.ipc.RemoteException"}}
Shouldn't httpfs overcome nn1 standby status and bring the file? Is that because of a wrong configuration, or is there any other reason?
My core-site.xml is
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
It looks like HttpFs is not High Availability aware yet. This could be due to the missing configurations required for the Clients to connect with the current Active Namenode.
Ensure the fs.defaultFS property in core-site.xml is configured with the correct nameservice ID.
If you have the below in hdfs-site.xml
<property>
<name>dfs.nameservices</name>
<value>mycluster</value>
</property>
then in core-site.xml, it should be
<property>
<name>fs.defaultFS</name>
<value>hdfs://mycluster</value>
</property>
Also configure the name of the Java class which will be used by the DFS Client to determine which NameNode is the currently Active and is serving client requests.
Add this property to hdfs-site.xml
<property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
Restart the Namenodes and HttpFs after adding the properties in all nodes.

HBase on docker NotServingRegionException because of hostname alisas

I am building a fully distributed hbase cluster with unmanaged zookeeper.
I pretty much used this example and install hbase on top of it: https://github.com/kiwenlau/hadoop-cluster-docker
Hadoop and hdfs works fine but I get this exception with hbase:
2016-09-05 06:27:12,268 INFO [hadoop-master:16000.activeMasterManager] zookeeper.MetaTableLocator: Failed verification of hbase:meta,,1 at address=hadoop-slave2,16020,1473052276351, exception=org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on hadoop-slave2.hadoopnet,16020,1473056813966
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2910)
This is bloking because any command I enter on the hbase shell will return the following error:
ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
The containers are runned using --net=hadoopnet
which is a network create as such:
docker network create --driver=bridge hadoopnet
The hbase webui is showing this:
Region Servers
ServerName Start time Version Requests Per Second Num. Regions
hadoop-slave1,16020,1473056814064 Mon Sep 05 06:26:54 UTC 2016 1.2.2 0 0
hadoop-slave1.hadoopnet,16020,1473056814064 Mon Sep 05 06:26:54 UTC 2016 Unknown 0 0
hadoop-slave2,16020,1473056813966 Mon Sep 05 06:26:53 UTC 2016 1.2.2 0 0
hadoop-slave2.hadoopnet,16020,1473056813966 Mon Sep 05 06:26:53 UTC 2016 Unknown 0 0
Total:4 2 nodes with inconsistent version 0 0
I should have only 2 regionservers but 2 strange hadoop-slave1.hadoopnet and hadoop-slave2.hadoopnet are added to the list.
When I look at zk using:
/usr/local/hbase/bin/hbase zkcli -server zk:2181 ls /hbase/rs
I only see my 2 regionserver: hadoop-slave1,16020,1473056814064 and hadoop-slave2,16020,1473056813966
Looking at the zookeeper.MetaTableLocator: Failed verification error I see that hadoop-slave2,16020,1473052276351 and hadoop-slave2.hadoopnet,16020,1473056813966 get mixed up.
here is my config on all server
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://hadoop-master:9000/hbase</value>
<description>The directory shared by region servers. Should be fully-qualified to include the filesystem to use. E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR</description>
</property>
<property>
<name>hbase.master</name>
<value>hdfs://hadoop-master:60000</value>
<description>The host and port that the HBase master runs at.</description>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)</description>
</property>
<property>
<name>hbase.master.info.port</name>
<value>60010</value>
<description>The UI interface of HBase master runs.</description>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>zk</value>
<description>string m_e_m_b_e_r_s is replaced by list of hosts separated by comma. Its generated by configure-slaves.sh on master node</description>
</property>
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>300</value>
</property>
<property>
<name>hbase.zookeeper.property.datadir</name>
<value>/tmp/zookeeper</value>
<description>location of storage of zookeeper data</description>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
</configuration>
I have the same problem, env as below :
hadoop 2.7.2
hbase 1.2.2
zookeeper 3.4.8
It came to my attention, that the hbase-1.2.2 included hadoop's jar as 2.5.1, zookeeper's jar as 3.4.6, I upgraded them to the version i'm using (hadoop & zookeeper), the error has gone, but still found [hostname].[docker-network] as a region-server, except this, everything is fine.

How to configure hadoop to use non-default port: "0.0.0.0: ssh: connect to host 0.0.0.0 port 22: Connection refused"

When I run start-dfs I get the below error and it looks like I need to tell hadoop to use a different port since that is what I require when I ssh into localhost. In other words the following works successfully: ssh -p 2020 localhost.
[Wed Jan 06 16:57:34 root#~]# start-dfs.sh
16/01/06 16:57:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: namenode running as process 85236. Stop it first.
localhost: datanode running as process 85397. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: ssh: connect to host 0.0.0.0 port 22: Connection refused
16/01/06 16:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
core-site.xml:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///hadoop/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///hadoop/hdfs/datanode</value>
</property>
</configuration>
If your Hadoop cluster nodes run sshd listening on a non-standard port, then it is possible to tell the Hadoop scripts to initiate ssh connections to that port. In fact, it's possible to customize any of the options passed to the ssh command.
This is controlled by an environment variable named HADOOP_SSH_OPTS. You can edit your hadoop-env.sh file and define it there. (By default this environment variable is not defined.)
For example:
export HADOOP_SSH_OPTS="-p 2020"

access hbase in IDE Eclipse , java.net.UnknownHostException

When I write the java code to access hbase in IDE Eclipse, the messages "java.net.UnknownHostException" are always been shown.But hbase shell works well.
I install the hadoop and hbase on a single linux node in pseudo distribution mode. And my hostname is yzd. Here are the /etc/hosts and hbase-site.xml:
/etc/hosts:
127.0.0.1 localhost yzd
hbase-site.xml:
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
Error message:
INFO [main] (HBaseRPC.java:117) - Using org.apache.hadoop.hbase.ipc.WritableRpcEngine for org.apache.hadoop.hbase.ipc.HMasterInterface
INFO [main] (HConnectionManager.java:596) - getMaster attempt 0 of 10 failed; retrying after sleep of 1000
java.net.UnknownHostException: unknown host: � 13846#yzdlocalhost
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.<init>(HBaseClient.java:224)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:954)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:816)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:141)
at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:174)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:295)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:272)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:324)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:579)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:94)
at com.hbasebook.hush.schema.SchemaManager.process(SchemaManager.java:126)
at com.hbasebook.hush.HushMain.main(HushMain.java:57)
Check the version of your local hbase matches the one you are using as a dependency in your pom. This should solve your issue. I was facing the same issue, I was using hbase in standalone mode. I hope this helps you.
First of all yzd is not host name, its domain name (You should prefer FQDN). Now this line
java.net.UnknownHostException: unknown host: � 13846#yzdlocalhost
clearly says that 13846#yzdlocalhost host is not there. Now you can do followings:
Use IP address instead of hostname in both hbase-site.xml and core-site.xml and check
Then use FQDN in etc/hosts file and tab-separate the values, now you can replace the IP with FQDN

Resources