I have been excluded datanode host "dn001" by "dfs_hosts_exclude.txt", and it works, how to also migrate datanode data from this "dn001" to other datanodes?
You shouldn't have to do anything. Hadoop's HDFS should re-replicate any data lost on your data node.
From HDFS Architecture - Data Disk Failure, Heartbeats and Re-Replication
Each DataNode sends a Heartbeat message to the NameNode periodically. A network partition can cause a subset of DataNodes to lose connectivity with the NameNode. The NameNode detects this condition by the absence of a Heartbeat message. The NameNode marks DataNodes without recent Heartbeats as dead and does not forward any new IO requests to them. Any data that was registered to a dead DataNode is not available to HDFS any more. DataNode death may cause the replication factor of some blocks to fall below their specified value. The NameNode constantly tracks which blocks need to be replicated and initiates replication whenever necessary. The necessity for re-replication may arise due to many reasons: a DataNode may become unavailable, a replica may become corrupted, a hard disk on a DataNode may fail, or the replication factor of a file may be increased.
Related
I have a question regarding hadoop hdfs blocks replication. Suppose a block is written on a datanode and the DFS has a replication factor 3, how long does it take for the namenode to replicate this block on other datanodes? Is it instantaneuos? If not, right after writing the block to a datanode suppose the disk on this datanode fails which cannot be recovered, does it mean the block is lost forever? And also how often does the namenode check for missing/corrupt blocks?
You may want to review this article which has a good description of hdfs writes. it should be immediate depending upon how busy the cluster is:
https://data-flair.training/blogs/hdfs-data-write-operation/
What happens if DataNode fails while writing a file in the HDFS?
While writing data to the DataNode, if DataNode fails, then the following actions take place, which is transparent to the client writing the data.
The pipeline gets closed, packets in the ack queue are then added to the front of the data queue making DataNodes downstream from the failed node to not miss any packet.
hdfs-site.xml:
dfs.replication value configured 3
Assuming that i set replication of an specific file to 2:
./bin/hadoop dfs -setrep -w 2 /path/to/file.txt
When NameNode receives heartbeat from DataNode,
Will NameNode consider as specified file
path/to/file.txt is in under replication as per the configured replication or not?
If not, how it 'll be?
First, I would like to attempt to restate your question for clarity, to make sure I understand:
Will the NameNode consider a file that has been manually set to a replication factor lower than the default (dfs.replication) to be under-replicated?
No. The NameNode stores the replication factor of each file separately in its metadata, even if the replication factor was not set explicitly by calling -setrep. By default, the metadata for each file will copy the replication factor as specified in dfs.replication (3 in your example). It may be overridden, such as by calling -setrep. When the NameNode checks if a file is under-replicated, it checks the exact replication factor stored in the metadata for that individual file, not dfs.replication. If the file's replication factor is 2, and there are 2 replicas of each of its blocks, then this is fine, and the NameNode will not consider it to be under-replicated.
Your question also makes mention of heartbeating from the DataNodes, which I think means you're interested in how interactions between the DataNodes and NameNodes relate to replication. There is also another form of communication between DataNodes and NameNodes called block reports. The block reports are the means by which DataNodes tell the NameNodes which block replicas they store. The NameNode analyzes block reports from all DataNodes to determine if a block is either under-replicated or over-replicated. If a block is under-replicated (e.g. replication factor is 2, but there is only one replica), then the NameNode schedules re-replication work so that another DataNode makes a copy of the replica. If a block is over-replicated (e.g. replication factor is 3, but there are 4 replicas), then the NameNode schedules one of the replicas to be deleted, and eventually one of the DataNodes will delete it locally.
One of the disk from my hadoop cluster datanode has become read only. I am not sure what caused this problem.
Will removing this volume from the datanode cause data lose ??
How to handle this if i am going to face data lose?
If your hadoop cluster was having a replication factor of more than 1 (by default it is 3 for a multi-node cluster), your data must have been replicated on multiple datanodes. You can check your replication factor value (dfs.replication) in hdfs-site.xml.
So now if you remove this read-only datanode from your cluster and you have a replication factor of more than 1, then you will not face any data loss. Because your cluster will have a corresponding replica on other datanode. To balance the replicas, under-replicated blocks will be handled by hdfs automatically and subsequently hdfs will be stable.
I have 10 data noes and 2 name nodes Hadoop cluster with replicates configured 3, I was wondering if one of data nodes goes down, will hadoop try to generate the lost replicates on the other alive nodes? or just do nothing(since still have 2 replicas left).
Add, what if the down data node come back after a while, can hadoop recognize the data on that node? Thanks!
will hadoop try to generate the lost replicates on the other alive nodes? or just do nothing(since still have 2 replicas left).
Yes, Hadoop will recognize it and make copies of that data on some other nodes. When Namenode stop receiving heart beats from the data nodes, it assumes that data node is lost. To keep the replication of the all the data to defined replication factor, it will make the copies on other data nodes.
Add, what if the down data node come back after a while, can hadoop recognize the data on that node?
Yes, when a data node comes back with all its data, Name node will remove/delete the extra copies of data. In the next heart beat to the data node, Name node will send the instruction to remove the extra data and free up the space on disk.
Snippet from Apache HDFS documentation:
Each DataNode sends a Heartbeat message to the NameNode periodically. A network partition can cause a subset of DataNodes to lose connectivity with the NameNode. The NameNode detects this condition by the absence of a Heartbeat message. The NameNode marks DataNodes without recent Heartbeats as dead and does not forward any new IO requests to them. Any data that was registered to a dead DataNode is not available to HDFS any more. DataNode death may cause the replication factor of some blocks to fall below their specified value. The NameNode constantly tracks which blocks need to be replicated and initiates replication whenever necessary. The necessity for re-replication may arise due to many reasons: a DataNode may become unavailable, a replica may become corrupted, a hard disk on a DataNode may fail, or the replication factor of a file may be increased.
The Namenode in the Hadoop architecture is a single point of failure.
How do people who have large Hadoop clusters cope with this problem?.
Is there an industry-accepted solution that has worked well wherein a secondary Namenode takes over in case the primary one fails ?
Yahoo has certain recommendations for configuration settings at different cluster sizes to take NameNode failure into account. For example:
The single point of failure in a Hadoop cluster is the NameNode. While the loss of any other machine (intermittently or permanently) does not result in data loss, NameNode loss results in cluster unavailability. The permanent loss of NameNode data would render the cluster's HDFS inoperable.
Therefore, another step should be taken in this configuration to back up the NameNode metadata
Facebook uses a tweaked version of Hadoop for its data warehouses; it has some optimizations that focus on NameNode reliability. Additionally to the patches available on github, Facebook appears to use AvatarNode specifically for quickly switching between primary and secondary NameNodes. Dhruba Borthakur's blog contains several other entries offering further insights into the NameNode as a single point of failure.
Edit: Further info about Facebook's improvements to the NameNode.
High Availability of Namenode has been introduced with Hadoop 2.x release.
It can be achieved in two modes - With NFS and With QJM
But high availability with Quorum Journal Manager (QJM) is preferred option.
In a typical HA cluster, two separate machines are configured as NameNodes. At any point in time, exactly one of the NameNodes is in an Active state, and the other is in a Standby state. The Active NameNode is responsible for all client operations in the cluster, while the Standby is simply acting as a slave, maintaining enough state to provide a fast failover if necessary.
Have a look at below SE questions, which explains complete failover process.
Secondary NameNode usage and High availability in Hadoop 2.x
How does Hadoop Namenode failover process works?
Large Hadoop clusters have thousands of data nodes and one name node. The probability of failure goes up linearly with machine count (all else being equal). So if Hadoop didn't cope with data node failures it wouldn't scale. Since there's still only one name node the Single Point of Failure (SPOF) is there, but the probability of failure is still low.
That sad, Bkkbrad's answer about Facebook adding failover capability to the name node is right on.