I am trying to archive HDFS data into a folder located at the path /archives using the command given below:
hadoop archive -archiveName zoo3.har -p /temp/test -r 3 /archives
But I get the following error:
File /archives/zoo3.har/_temporary/1/_temporary/attempt_1498217315045_0006_m_000000_2/part-0
could only be replicated to 0 nodes instead of minReplication (=1).
There are 6 datanode(s) running and no node(s) are excluded in this
operation.
Cluster Configuration:
6 Data Nodes (all active)
Each with 5 TB disk and 64 GB RAM
HDP - 2.4
Related
I am having CDH virtual box running on my windows 10. I am running simple talend job, which has only component to put file on HDFS (tHDFSPut) from windows to HDFS which is located in virtual box. But when I run the job the file is created on HDFS but it is empty.
I am getting following error,
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/cloudera/test/Input.xml could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1716)
I checked, namenode is healthy and it is having sufficient disk space available, also I can able to access it outside the virtual box using localhost:50070
Need help!
I am trying to put a csv file from my local windows machine to HDFS using Talend 6.3
I have 4 node cluster ( all linux servers from azure):
1 Namenode and 3 Datanodes.
I am getting following error while running:
"Exception in component tHDFSPut_1
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/root/5/source.csv could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and 3 node(s) are excluded in this operation."
File is getting created in HDFS but it is empty file.
Note: I am writing my Namenode IP address in NameNode URL
I tried same with sandbox. Same error was there.
Update:
I created a complete new cluster(1 namenode and 3 datanodes). Still I am getting same error.
Everything is running and is is in green (in Ambari). and I am able to put file through hadoop fs -put. Might be some problem in talend?
I have fired up a Spark Cluster on Amazon EC2 containing 1 master node and 2 servant nodes that have 2.7gb of memory each
However when I tried to put a file of 3 gb on to the HDFS through the code below
/root/ephemeral-hdfs/bin/hadoop fs -put /root/spark/2GB.bin 2GB.bin
it returns the error, "/user/root/2GB.bin could only be replicated to 0 nodes, instead of 1". fyi, I am able to upload files of smaller size but not when it exceeds a certain size (about 2.2 gb).
If the file exceeds the memory size of a node, wouldn't it will be split by Hadoop to the other node?
Edit: Summary of my understanding of the issue you are facing:
1) Total HDFS free size is 5.32 GB
2) HDFS free size on each node is 2.6GB
Note: You have bad blocks (4 Blocks with corrupt replicas)
The following Q&A mentions similar issues:
Hadoop put command throws - could only be replicated to 0 nodes, instead of 1
In that case, running JPS showed that the datanode are down.
Those Q&A suggest a way to restart the data-node:
What is best way to start and stop hadoop ecosystem, with command line?
Hadoop - Restart datanode and tasktracker
Please try to restart your data-node, and let us know if it solved the problem.
When using HDFS - you have one shared file system
i.e. all nodes share the same file system
From your description - the current free space on the HDFS is about 2.2GB , while you tries to put there 3GB.
Execute the following command to get the HDFS free size:
hdfs dfs -df -h
hdfs dfsadmin -report
or (for older versions of HDFS)
hadoop fs -df -h
hadoop dfsadmin -report
I have a 4-node hadoop cluster. It is running fine.
I temporarily stop hadoop cluster using:
stop-dfs.sh
stop-yarn.sh
When I restart it using:
start-dfs.sh
start-yarn.sh
all jps(hadoop processes) are running fine on all the 4 nodes,
but it is showing the following error log on running map-reduce job:
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/hduser4/QuasiMonteCarlo_14 21995096158_792628557/in/part0 could only be replicated to 0 nodes instead of minReplication (=1).
There are 0 datanode(s) running and no node(s) are excluded in this operation.
An option for the problem above is to reformat HDFS, but it will delete my existing data, which is not a proper solution for a production system.
I've installed CDH4 on a ubuntu 12 LTS server successfully in the amazon cloud (1 server). I used Cloudera Manager free edition to install the software and had no errors).
I have a program that uses the java API to load a file from my home computer to HDFS in the cloud. I would like to know why this program fails and how to fix it.
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://node01:8020");
FileSystem fs = FileSystem.get(conf);
Path targetPath = new Path("/users/<username>/myfile.txt");
Path sourcePath = new Path("/home/<username>/myfile.txt");
fs.copyFromLocalFile(false,true,sourcePath,targetPath);
I get the following error (namenode log):
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/<username>/myfile.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1322)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2170)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:471)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:297)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44080)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
Then i upload my code to the cloud and run the code locally (uploading a file from the local fs to hdfs) there aren't any errors. It only happens when I run the code from my personal computer...
When i use the cli 'hadoop fs -put' command on my cloud server, I get no errors when writing to hdfs. I can also upload files using Hue. I've done some reading and found that this problem occurs when there isn't enough disk space, but I have plenty for both dfs and non-dfs (see report below). I can successfully read the hadoop filesystem with the java API from my home computer, and I can even connect and read/write from HBase using the API. All ports are open to my IP on this server. File permissions have been checked. After the program fails, I see the file I tried to upload in hdfs , but the contents are blank (similar to this post: https://groups.google.com/a/cloudera.org/forum/?fromgroups=#!topic/cdh-user/XWA-3H0ekYY )
here is the output from hdfs dfsadmin -report
Configured Capacity: 95120474112 (88.59 GB)
Present Capacity: 95120474112 (88.59 GB)
DFS Remaining: 95039008768 (88.51 GB)
DFS Used: 81465344 (77.69 MB)
DFS Used%: 0.09%
Under replicated blocks: 177
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)
Live datanodes:
Name: privateip:port (node01)
Hostname: node01
Rack: /default
Decommission Status : Normal
Configured Capacity: 95120474112 (88.59 GB)
DFS Used: 81465344 (77.69 MB)
Non DFS Used: 0 (0 KB)
DFS Remaining: 95039008768 (88.51 GB)
DFS Used%: 0.09%
DFS Remaining%: 99.91%
Last contact: Sun Jan 27 03:01:53 UTC 2013
I've resolved the problem-
I was connecting to hadoop from my home machine not on the hadoop local network. Apparently when you do this, the namenode tells my home machine to write to the datanode using the datanode's private IP. Not being on the same network, my home machine coulnd't connect to the datanode creating this error.
I resolved the problem by creating a VPN connection from my home network to the hadoop network and now everything work.