Environment
Hadoop : 0.20.205.0
Number of machines in cluster : 2 nodes
Replication : set to 1
DFS Block size : 1MB
I put a 7.4MB file into HDFS using put command. I run fsck command to check the blocks distribution of the file among the datanodes. I see that all the 8 blocks of the file are going to only one node. This affects the load distribution and only one node always get used while running mapred tasks.
Is there a way that I can distribute the files to more than one datanode?
bin/hadoop dfsadmin -report
Configured Capacity: 4621738717184 (4.2 TB)
Present Capacity: 2008281120783 (1.83 TB)
DFS Remaining: 2008281063424 (1.83 TB)
DFS Used: 57359 (56.01 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 2 (6 total, 4 dead)
Name: 143.215.131.246:50010
Decommission Status : Normal
Configured Capacity: 2953506713600 (2.69 TB)
DFS Used: 28687 (28.01 KB)
Non DFS Used: 1022723801073 (952.49 GB)
DFS Remaining: 1930782883840(1.76 TB)
DFS Used%: 0%
DFS Remaining%: 65.37%
Last contact: Fri Jul 18 10:31:51 EDT 2014
bin/hadoop fs -put /scratch/rkannan3/hadoop/test/pg20417.txt /user/rkannan3
bin/hadoop fs -ls /user/rkannan3
Found 1 items
-rw------- 1 rkannan3 supergroup 7420270 2014-07-18 10:40 /user/rkannan3/pg20417.txt
bin/hadoop fsck /user/rkannan3 -files -blocks -locations
FSCK started by rkannan3 from /143.215.131.246 for path /user/rkannan3 at Fri Jul 18 10:43:13 EDT 2014
/user/rkannan3 <dir>
/user/rkannan3/pg20417.txt 7420270 bytes, 8 block(s): OK <==== All the 8 blocks in one DN
0. blk_3659272467883498791_1006 len=1048576 repl=1 [143.215.131.246:50010]
1. blk_-5158259524162513462_1006 len=1048576 repl=1 [143.215.131.246:50010]
2. blk_8006160220823587653_1006 len=1048576 repl=1 [143.215.131.246:50010]
3. blk_4541732328753786064_1006 len=1048576 repl=1 [143.215.131.246:50010]
4. blk_-3236307221351862057_1006 len=1048576 repl=1 [143.215.131.246:50010]
5. blk_-6853392225410344145_1006 len=1048576 repl=1 [143.215.131.246:50010]
6. blk_-2293710893046611429_1006 len=1048576 repl=1 [143.215.131.246:50010]
7. blk_-1502992715991891710_1006 len=80238 repl=1 [143.215.131.246:50010]
If you want to have distribution on file level use at least a replication factor of 2. The first replica is always placed where the writer is located
(see the introduction paragraph in http://waset.org/publications/16836/optimizing-hadoop-block-placement-policy-and-cluster-blocks-distribution); and normally one file has only one writer, so the first replica of several blocks of a file will always be on that node. You probably don't want to change that behaviour, because you want to have the option available to increase the minimum split size when you want to avoid spawning too many mappers without losing the data locality for mappers.
You must use Hadoop balancer command. Details below. Tutorials link
Balancer
Runs a cluster balancing utility. You can simply press Ctrl-C to stop the rebalancing process. Please find more details here
Usage: hadoop balancer [-threshold <threshold>]
-threshold <threshold> Percentage of disk capacity. This overwrites the default threshold.
Related
I am unable to delete corrupt files present in my HDFS. Namenode has run into Safe mode. Total number of blocks are 980, out of which 978 have reported. When I run the following command,
sudo -u hdfs hdfs dfsadmin -report
The report generated is,
Safe mode is ON
Configured Capacity: 58531520512 (54.51 GB)
Present Capacity: 35774078976 (33.32 GB)
DFS Remaining: 32374509568 (30.15 GB)
DFS Used: 3399569408 (3.17 GB)
DFS Used%: 9.50%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (1):
Name: 10.0.2.15:50010 (quickstart.cloudera)
Hostname: quickstart.cloudera
Decommission Status : Normal
Configured Capacity: 58531520512 (54.51 GB)
DFS Used: 3399569408 (3.17 GB)
Non DFS Used: 19777388544 (18.42 GB)
DFS Remaining: 32374509568 (30.15 GB)
DFS Used%: 5.81%
DFS Remaining%: 55.31%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 14 10:39:58 IST 2017
And for the following command when executed,
sudo -u hdfs hdfs fsck /
The output is,
Connecting to namenode via http://quickstart.cloudera:50070/fsck?ugi=hdfs&path=%2F
FSCK started by hdfs (auth:SIMPLE) from /10.0.2.15 for path / at Tue Nov 14 10:41:25 IST 2017
/hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.default.1509701903728: CORRUPT blockpool BP-1914853243-127.0.0.1-1500467607052 block blk_1073743141
/hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.default.1509701903728: MISSING 1 blocks of total size 83 B..
/hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.meta.1509701932269.meta: CORRUPT blockpool BP-1914853243-127.0.0.1-1500467607052 block blk_1073743142
/hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.meta.1509701932269.meta: MISSING 1 blocks of total size 83 B
Status: CORRUPT
Total size: 3368384392 B (Total open files size: 166 B)
Total dirs: 286
Total files: 966
Total symlinks: 0 (Files currently being written: 3)
Total blocks (validated): 980 (avg. block size 3437126 B) (Total open file blocks (not validated): 2)
********************************
UNDER MIN REPL'D BLOCKS: 2 (0.20408164 %)
dfs.namenode.replication.min: 1
CORRUPT FILES: 2
MISSING BLOCKS: 2
MISSING SIZE: 166 B
CORRUPT BLOCKS: 2
********************************
Minimally replicated blocks: 978 (99.79592 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 1
Average block replication: 0.9979592
Corrupt blocks: 2
Missing replicas: 0 (0.0 %)
Number of data-nodes: 1
Number of racks: 1
FSCK ended at Tue Nov 14 10:41:26 IST 2017 in 774 milliseconds
The filesystem under path '/' is CORRUPT
Can anyone please help in either correcting the corrupted blocks, (or) deleting them? Thanks in advance.
As it's said that Namenode is in Safe mode, first turn it off.
hdfs dfsadmin -safemode leave
Then execute either of commands
hdfs fsck / | egrep -v '^\.+$' | grep -v replica | grep -v Replica
or
hdfs fsck hdfs://quickstart.cloudera:50070/ | egrep -v '^\.+$' | grep -v replica | grep -v Replica
The output would be somewhat similar to
/path/to/filename.fileextension: CORRUPT blockpool BP-1016133662-10.29.100.41-1415825958975 block blk_1073904305
/path/to/filename.fileextension: MISSING 1 blocks of total size 15620361 B
In your case corrupted files are already listed. So execute below commands
hdfs dfs -rm /hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.default.1509701903728
hdfs dfs -rm /hbase/oldWALs/quickstart.cloudera%2C60020%2C1509698296866.meta.1509701932269.meta
And don't enter into Safemode. Just continue working.
Yippee!!
I faced a weird situation where i'm getting different results from hdfs dfs -du command and what i see in cloudera manager UI, i read about the differences between the 2 but didn't any clue that could help me to catch the issue and solved it.
I also deleted all the snapshots and disallowed them, but the storage didn't change.
Below is the output :
[cloudera-scm#roor-chc101 root]$ hdfs dfs -du -h -s .
2.3 G 5.8 G .
[cloudera-scm#roor-chc101 root]$ hdfs dfs -du -h -s /
250.3 T 749.3 T
Cloudera manager UI
I also checked the hdfs dfsadmin -report which shows the same results as the UI
Configured Capacity: 1.54 PB
DFS Used: 897.77 TB
Non DFS Used: 2.98 GB
DFS Remaining: 682.27 TB
DFS Used%: 56.82%
DFS Remaining%: 43.18%
Block Pool Used: 897.77 TB
Block Pool Used%: 56.82%
DataNodes usages% (Min/Median/Max/stdDev): 11.17% / 58.94% / 69.35% / 13.31%
Live Nodes 45 (Decommissioned: 0)
Dead Nodes 0 (Decommissioned: 0)
Decommissioning Nodes 0
Total Datanode Volume Failures 0 (0 B)
Number of Under-Replicated Blocks 0
Number of Blocks Pending Deletion 0
Block Deletion Start Time 8/14/2017, 10:57:30 AM
i added hdfs dfs -du -h -s . and hdfs dfs -du -h -s / just to show nothing that i may missed which could be under /user/cloudera-scm.
The cloudera manager UI shows the same value as the hdfs dfsadmin -report.
BTW, i solved the issue where i find /tmp/logs/cloudera-scm used 150T and still intersting why this volume wasn't taken into consideration when i ran hdfs dfs -du -h -s
Is there a way to find which datanodes a particular hdfs file is stored on, or a list of blocks that store an hdfs file?
For example, if I have hdfs://user/person/file.csv,
Is there a way to find a list of ext4 paths corresponding to the blocks that make up this file on the datanodes?
Yes, you can find out the location of blocks which are stored on different datanodes in HDFS. Here is the command:
hdfs fsck /user/hduser/file.txt -files -blocks -locations
This will give you all the information related to individual blocks created for file: "/user/hduser/file.txt". Output generally looks like this:
[hduser#node001 ~]$ hdfs fsck /user/hduser/file.txt -files -blocks -locations
Connecting to namenode via http://node001.morado.com:50070
FSCK started by hduser (auth:SIMPLE) from /192.168.2.169 for path /user/hduser/file.txt at Mon Jul 11 23:14:27 PDT 2016
/user/hduser/file.txt 1073839694 bytes, 9 block(s): OK
0. BP-778802867-192.168.2.147-1465886958278:blk_1080847742_7107323 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-25d2b73a-2dc2-48c1-9aad-f0f5ca8d302a,DISK], DatanodeInfoWithStorage[192.168.2.147:50010,DS-293a7f8d-ad31-4bc1-98d8-0c0822eda305,DISK], DatanodeInfoWithStorage[192.168.2.20:50010,DS-8efb7a6e-08f0-4f2d-aee2-bc5a102277bd,DISK]]
1. BP-778802867-192.168.2.147-1465886958278:blk_1080847748_7107329 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-6881b609-1473-48d5-a07c-f111e0bdcf2f,DISK], DatanodeInfoWithStorage[192.168.2.15:50010,DS-060c75ff-5632-4f6f-a73b-fb2a68927c63,DISK], DatanodeInfoWithStorage[192.168.2.147:50010,DS-3e108776-d3bd-4b84-b68a-59e1ca755331,DISK]]
2. BP-778802867-192.168.2.147-1465886958278:blk_1080847753_7107334 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-25d2b73a-2dc2-48c1-9aad-f0f5ca8d302a,DISK], DatanodeInfoWithStorage[192.168.2.177:50010,DS-b7a33931-8917-4fe2-b2ec-2e4c3d5b6b01,DISK], DatanodeInfoWithStorage[192.168.2.135:50010,DS-5efb0813-7e4e-4d27-8fa4-7f8f3b2e6e3c,DISK]]
3. BP-778802867-192.168.2.147-1465886958278:blk_1080847760_7107341 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-6881b609-1473-48d5-a07c-f111e0bdcf2f,DISK], DatanodeInfoWithStorage[192.168.2.20:50010,DS-b8a5ceaf-6953-4842-8930-29b286ccb7cf,DISK], DatanodeInfoWithStorage[192.168.2.134:50010,DS-c6418fbb-6e30-447e-b507-bf19e0f28fd1,DISK]]
4. BP-778802867-192.168.2.147-1465886958278:blk_1080847764_7107345 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-95636645-c59e-4bca-8478-c15b3c16d514,DISK], DatanodeInfoWithStorage[192.168.2.147:50010,DS-293a7f8d-ad31-4bc1-98d8-0c0822eda305,DISK], DatanodeInfoWithStorage[192.168.2.20:50010,DS-8efb7a6e-08f0-4f2d-aee2-bc5a102277bd,DISK]]
5. BP-778802867-192.168.2.147-1465886958278:blk_1080847768_7107349 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-6881b609-1473-48d5-a07c-f111e0bdcf2f,DISK], DatanodeInfoWithStorage[192.168.2.20:50010,DS-276bd2c8-ee3d-4cd3-b655-17a83917c45b,DISK], DatanodeInfoWithStorage[192.168.2.134:50010,DS-5e128658-c876-46df-b10e-5962baf73db2,DISK]]
6. BP-778802867-192.168.2.147-1465886958278:blk_1080847772_7107353 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-fa57d5e9-a187-4856-8bf2-6933e63b3afe,DISK], DatanodeInfoWithStorage[192.168.2.135:50010,DS-f5f1e2a0-186b-4c70-844f-7e7ebe389f50,DISK], DatanodeInfoWithStorage[192.168.2.15:50010,DS-8cfb8ffb-77b6-40bb-930e-81c7198166ad,DISK]]
7. BP-778802867-192.168.2.147-1465886958278:blk_1080847776_7107357 len=134217728 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-95636645-c59e-4bca-8478-c15b3c16d514,DISK], DatanodeInfoWithStorage[192.168.2.15:50010,DS-060c75ff-5632-4f6f-a73b-fb2a68927c63,DISK], DatanodeInfoWithStorage[192.168.2.147:50010,DS-3e108776-d3bd-4b84-b68a-59e1ca755331,DISK]]
8. BP-778802867-192.168.2.147-1465886958278:blk_1080847780_7107361 len=97870 repl=3 [DatanodeInfoWithStorage[192.168.2.169:50010,DS-25d2b73a-2dc2-48c1-9aad-f0f5ca8d302a,DISK], DatanodeInfoWithStorage[192.168.2.15:50010,DS-de3cffa6-cef4-4f47-9bbf-5f44214b3a5a,DISK], DatanodeInfoWithStorage[192.168.2.177:50010,DS-847ec520-bc14-4ca4-af94-21140a3b20f6,DISK]]
Status: HEALTHY
Total size: 1073839694 B
Total dirs: 0
Total files: 1
Total symlinks: 0
Total blocks (validated): 9 (avg. block size 119315521 B)
Minimally replicated blocks: 9 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 7
Number of racks: 1
FSCK ended at Mon Jul 11 23:14:27 PDT 2016 in 1 milliseconds
The filesystem under path '/user/hduser/file.txt' is HEALTHY
Look for the information after "repl", usually starting with "DatanodeInfoWithStorage" tag. It gives the required information about datanode locations.
I've a file on hdfs in the path 'test/test.txt' which is 1.3G
output of ls and du commands is:
hadoop fs -du test/test.txt -> 1379081672 test/test.txt
hadoop fs -ls test/test.txt ->
Found 1 items
-rw-r--r-- 3 testuser supergroup 1379081672 2014-05-06 20:27 test/test.txt
I want to run a mapreduce job on this file but when i start the mapreduce job on this file the job fails with the following error:
hadoop jar myjar.jar test.TestMapReduceDriver test output
14/05/29 16:42:03 WARN mapred.JobClient: Use GenericOptionsParser for parsing the
arguments. Applications should implement Tool for the same.
14/05/29 16:42:03 INFO input.FileInputFormat: Total input paths to process : 1
14/05/29 16:42:03 INFO mapred.JobClient: Running job: job_201405271131_9661
14/05/29 16:42:04 INFO mapred.JobClient: map 0% reduce 0%
14/05/29 16:42:17 INFO mapred.JobClient: Task Id : attempt_201405271131_9661_m_000004_0, Status : FAILED
java.io.IOException: Cannot obtain block length for LocatedBlock{BP-428948818-namenode-1392736828725:blk_-6790192659948575136_8493225; getBlockSize()=36904392; corrupt=false; offset=1342177280; locs=[datanode4:50010, datanode3:50010, datanode1:50010]}
at org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:319)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:263)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:205)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:198)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1117)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:249)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:82)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:746)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:83)
at org.apache.hadoop.mapred.Ma`
i tried the following commands:
hadoop fs -cat test/test.txt gives the following error
cat: Cannot obtain block length for LocatedBlock{BP-428948818-10.17.56.16-1392736828725:blk_-6790192659948575136_8493225; getBlockSize()=36904392; corrupt=false; offset=1342177280; locs=[datanode3:50010, datanode1:50010, datanode4:50010]}
additionally i can't copy the file hadoop fs -cp test/test.txt tmp gives same error:
cp: Cannot obtain block length for LocatedBlock{BP-428948818-10.17.56.16-1392736828725:blk_-6790192659948575136_8493225; getBlockSize()=36904392; corrupt=false; offset=1342177280; locs=[datanode1:50010, datanode3:50010, datanode4:50010]}
output of the hdfs fsck /user/testuser/test/test.txt command:
Connecting to namenode via `http://namenode:50070`
FSCK started by testuser (auth:SIMPLE) from /10.17.56.16 for path
/user/testuser/test/test.txt at Thu May 29 17:00:44 EEST 2014
Status: HEALTHY
Total size: 0 B (Total open files size: 1379081672 B)
Total dirs: 0
Total files: 0 (Files currently being written: 1)
Total blocks (validated): 0 (Total open file blocks (not validated): 21)
Minimally replicated blocks: 0
Over-replicated blocks: 0
Under-replicated blocks: 0
Mis-replicated blocks: 0
Default replication factor: 3
Average block replication: 0.0
Corrupt blocks: 0
Missing replicas: 0
Number of data-nodes: 5
Number of racks: 1
FSCK ended at Thu May 29 17:00:44 EEST 2014 in 0 milliseconds
The filesystem under path /user/testuser/test/test.txt is HEALTHY
by the way i can see the content of the test.txt file from the web browser.
hadoop version is: Hadoop 2.0.0-cdh4.5.0
I got the same issue with you and I fixed it by the following steps.
There are some files that opened by flume but never closed (I am not sure about your reason).
You need to find the name of the opened files by the command:
hdfs fsck /directory/of/locked/files/ -files -openforwrite
You can try to recover files as command:
hdfs debug recoverLease -path <path-of-the-file> -retries 3
Or removing them by the command:
hdfs dfs -rmr <path-of-the-file>
I had the same error, but it was not due to the full disk problem, and I think the inverse, where there were files and blocks referenced by in the namenode that did not exist on any datanodes.
Thus, hdfs dfs -ls shows the files, but any operation on them fails, e.g. hdfs dfs -copyToLocal.
In my case, the hard part was isolating which files were listed but corrupted, as they existed in a tree having thousands of files. Oddly, hdfs fsck /path/to/files/ did not report any problems.
My solution was:
Isolate the location using copyToLocal which resulted in copyToLocal: Cannot obtain block length for LocatedBlock{BP-1918381527-10.74.2.77-1420822494740:blk_1120909039_47667041; getBlockSize()=1231; corrupt=false; offset=0; locs=[10.74.2.168:50010, 10.74.2.166:50010, 10.74.2.164:50010]} for several files
Get a list of the local directories using ls -1 > baddirs.out
get rid of the local files from the first copyToLocal
use for files incat baddirs.out;do echo $files; hdfs dfs -copyToLocal $files This will produce a list of directories checks, and errors where files are found.
get rid of the local files again, and now get lists of files from each affected subdirectory. Use that as input to a file-by-file copyToLocal, at which point you can echo each file as it's copied, then see where the error occurs.
use hdfs dfs -rm <file> for each file.
Confirm you got 'em all be removing all local files again, and using the original copyToLocal on the top level directory where you had problems.
A simple two hour process!
You are having some corrupted files with no blocks on datanode but an entry in namenode. Best to follow this:
https://stackoverflow.com/a/19216037/812906
According to this this may be produced by a full disk problem. I came across the same problem recently with an old file and checking my servers metrics it effectively was a full disk problem during the creation of that file. Most solutions just claim to delete the file and prey for it not happening again.
i want to use Centralized Cache in hadoop-2.3.
here is my steps. (10 nodes, every node 6g memory)
1.my file(45M) to be cached
[hadoop#Master ~]$ hadoop fs -ls /input/pics/bundle
Found 1 items
-rw-r--r-- 1 hadoop supergroup 47185920 2014-03-09 19:10 /input/pics/bundle/bundle.chq
2.create cache pool
[hadoop#Master ~]$ hdfs cacheadmin -addPool myPool -owner hadoop -group supergroup
Successfully added cache pool myPool.
[hadoop#Master ~]$ hdfs cacheadmin -listPools -stats
Found 1 result.
NAME OWNER GROUP MODE LIMIT MAXTTL BYTES_NEEDED BYTES_CACHED BYTES_OVERLIMIT FILES_NEEDED FILES_CACHED
myPool hadoop supergroup rwxr-xr-x unlimited never 0 0 0 0 0
3.addDirective
[hadoop#Master ~]$ hdfs cacheadmin -addDirective -path /input/pics/bundle/bundle.chq -pool myPool -force -replication 3
Added cache directive 2
4.listDirectives
[hadoop#Master ~]$ hdfs cacheadmin -listDirectives -stats -path /input/pics/bundle/bundle.chq -pool myPool
Found 1 entry
ID POOL REPL EXPIRY PATH BYTES_NEEDED BYTES_CACHED FILES_NEEDED FILES_CACHED
2 myPool 3 never /input/pics/bundle/bundle.chq 141557760 0 1 0
the BYTES_NEEDED is right, but BYTES_CACHED is zero. It seems that the size has been calculated but the cache action which puts file into memory has not been done.So how to cache my file into memory.
Thank you very much.
There were a bunch of bugs we fixed in Hadoop 2.3. I would recommend using at least Hadoop 2.4 to use HDFS caching.
To get into more detail I would need to see the log messages.
Including the output of hdfs dfsadmin -report would also be useful, as well as ensuring that you have followed the setup instructions here (namely, increasing the ulimit and setting dfs.datanode.max.locked.memory):
http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html