DataNodes can't talk to NameNode - hadoop

Question

Set up a Hadoop cluster of 3 nodes. One of them got both NameNode and DataNode roles while other two are just DataNodes.
I started all nodes and services but in summary it shows only one of DataNodes's status is live. Status of other nodes are not even showing.
My question is what is the difference between being started and being live? And why other nodes don't have a status at all?
I guess the issue is datanodes can't talk to namenode. So as Azwaw pointed out, I checked /etc/hosts file. It was like this:
127.0.0.1 nnode.domain nnode localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.1.4.212 nnode.domain nnode
192.1.5.124 dnode02.domain dnode02
192.1.5.125 dnode01.domain dnode01
I changed first line to:
127.0.0.1 localhost.localdomain localhost localhost4 localhost4.localdomain4
Now I can establish connection with nnode.domain:50070, however errors at datanode side changed. Here log piece from datanode:
2015-05-15 10:08:21,721 ERROR datanode.DataNode (DataXceiver.java:run(253)) - dnode01.domain:50010:DataXceiver error processing unknown operation src: /127.0.0.1:49000 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:212)
at java.lang.Thread.run(Thread.java:745)
2015-05-15 10:08:23,670 INFO datanode.DataNode (BPServiceActor.java:register(782)) - Block pool BP-2116866246-127.0.0.1-1431441630609 (Datanode Uuid null) service to nnode.domain/192.1.4.212:8020 beginning handshake with NN
2015-05-15 10:08:23,674 ERROR datanode.DataNode (BPServiceActor.java:run(840)) - Initialization failed for Block pool BP-2116866246-127.0.0.1-1431441630609 (Datanode Uuid null) service to nnode.domain/192.1.4.212:8020 Datanode denied communication with namenode because hostname cannot be resolved (ip=192.1.4.1, hostname=192.1.4.1): DatanodeRegistration(0.0.0.0, datanodeUuid=7f1be518-1255-4a6a-b31c-22be5dc47673, infoPort=50075, ipcPort=8010, storageInfo=lv=-56;cid=CID-51d1dfd0-9376-44a7-b581-c14eec95fd74;nsid=450599258;c=0)
at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:887)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:5282)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:1082)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:92)
at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:26378)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
This is odd, there is no host with 192.1.4.1 IP address. Why would datanodes try to connect 192.1.4.1?
Unresolved datanode registration: hostname cannot be resolved (ip=192.1.4.1, hostname=192.1.4.1)

Answer 1

"Datanodes 3/3 started" means 3 datanodes process running
Datanodes status "1 live / 0 Dead / 0 Decommissioning" means your namenode is able to communicate with one node.
It seems to be a network problem (make sure HDFS ports are open on your firewall). The alive Datanode is probably on same machine as your Namenode.

Answer 2

Moving NameNode to same network with DataNodes solved the problem.
DataNodes are in 192.1.5.* network.
NameNode was in 192.1.4.* network.
After moving NameNode to 192.1.5.* did the trick for my case.