In hadoop definitive guide :
a 1 MB file stored with a block size of 128 MB uses 1 MB of disk space, not128 MB.
what does this mean ?
does it use 1MB of size in a block of 128MB or 1MB is used and reamining 127MB is free to occupy by some other file ?
This is often a misconception about HDFS - the block size is more about how a single file is split up / partitioned, not about some reserved part of the file system.
Behind the schemes, each block is stored on the DataNodes underlying files system as a plain file (and an associated checksum). If you look into the data node folder on your disks you should be able to find the file (if you know the file's block ID and data node allocations - which you can discover from the NameNode Web UI).
So back to your question, a 1MB file with a block size of 16MB/32MB/128MB/512MB/1G/2G (you get the idea) will still only be a 1MB file on the data nodes disk. The difference between the block size and the amount of data stored in that block is then free for the underlying file system to use as it sees fit (by HDFS, or something else).
Hadoop Block size is Hadoop Storage Concept. Every Time When you store a File in Hadoop it will divided into the block sizes and based on the replication factor and data locality it will be distributed over the cluster.
For Details you can find my answer here
Small files and HDFS blocks
Related
I'm working on storage aspect of Hadoop and exploring on know how ORC files get stored on HDFS block.
In HDFS, a file is composed of blocks. One block cannot hold multiple files.
Two ORC files, each with 1MB, will need a block per file.
If you are concerned about the actual disk storage it might consume, it will be 2MB only. Though the blocks are 128MB, the disk storage it determined by the size of the actual file/block.
This is with reference to the question : Small files and HDFS blocks where the answer quotes Hadoop: The Definitive Guide:
Unlike a filesystem for a single disk, a file in HDFS that is smaller than a single block does not occupy a full block’s worth of underlying storage.
Which I completely agree with because as per my understanding, blocks are just a way for the namenode to map which piece of file is where in the entire cluster. And since HDFS is an abstraction over our regular filesystems, there is no way a 140 MB will consume 256 MB of space on HDFS if the block size is 128MB, or in other words, the remaining space in the block will not get wasted.
However, I stumbled upon another answer here in Hadoop Block size and file size issue which says:
There are limited number of blocks available dependent on the capacity of the HDFS. You are wasting blocks as you will run out of them before utilizing all the actual storage capacity.
Does that mean if I have 1280 MB of HDFS storage and I try to load 11 files with size 1 MB each ( considering 128 MB block size and 1 replication factor per block ), the HDFS will throw an error regarding the storage?
Please correct if I am assuming anything wrong in the entire process. Thanks!
No. HDFS will not throw error because
1280 MB of storage limit is not exhausted.
11 meta entries won't cross memory limits on the namenode.
For example, say we have 3GB of memory available on namenode. Namenode need to store meta entries for each file, each block. Each of this entries take approx. 150 bytes. Thus, you can store roughly max. 1 million files with each having one block. Thus, even if you have much more storage capacity, you will not be able to utilize it fully if you have multiple small files reaching the memory limit of namenode.
But, specific example mentioned in the question does not reach this memory limit. Thus, there should not be any error.
Consider, hypothetical scenario having available memory in the namenode is just 300 bytes* 10. In this case, it should give an error for request to store 11th block.
References:
http://blog.cloudera.com/blog/2009/02/the-small-files-problem/
https://www.mail-archive.com/core-user#hadoop.apache.org/msg02835.html
I have few questions regarding Hadoop architecture
In Mapreduce can we dynamically modify the block size and no of mappers,if so how do we do?
Hows does the block gets created in HDFS. For example the hadoop framework is installed on say redhat linux machine. The default block size of linux filesystem is 4k. Is the HDFS block a logical wrapper on the 4k blocks or how does a block gets created. also is it parallel or sequential? because for example a file has only 32 MB since the block size is 64 MB. Is the remaining 32 Mb reusable?
I want to see the location(data node) of all the blocks of particular file I just copied to the HDFS. Is there any command to do that from a single location?
If I move the video file to HDFS, how does the block allocation happen for this video file
In Mapreduce can we dynamically modify the block size and no of mappers?
I assume that you are looking for HDFS file system.
HDFS is distributed storage system and Mapreduce is distributed processing framework.
HDFS block size can be changed with hdfs-site.xml
Have a look at documentation page for various HDFS configurations.
dfs.blocksize
134217728 ( default value)
The default block size for new files, in bytes. You can use the following suffix (case insensitive): k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.), Or provide complete size in bytes (such as 134217728 for 128 MB).
Relate SE question:
How to set data block size in Hadoop ? Is it advantage to change it?
Hows does the block gets created in HDFS. For example the hadoop framework is installed on say redhat linux machine. The default block size of linux filesystem is 4k. Is the HDFS block a logical wrapper on the 4k blocks or how does a block gets created. also is it parallel or sequential? because for example a file has only 32 MB since the block size is 64 MB. Is the remaining 32 Mb reusable?
Remaining 32 MB is re-usable.
Have a look at this SE question for HDFS block write operation :
Hadoop file write
I want to see the location(data node) of all the blocks of particular file I just copied to the HDFS. Is there any command to do that from a single location?
hadoop fsck /path/to/file -files -blocks
Related SE question:
Viewing the number of blocks for a file in hadoop
If I move the video file to HDFS, how does the block allocation happen for this video file?
Number of blocks = File size in MB / DFS block size in MB
Once number of blocks have been identified, those blocks will be written as explained in Hadoop file write question.
Few more good questions:
Hadoop chunk size vs split vs block size
How hadoop decides how many nodes will do map and reduce tasks
My file size is 65MB and default hdfs block size(64MB), then how many 64MB blocks will be allotted to my file?
Is it like 1-64MB block, 1-1MB block or 2-64MB blocks? If it is 2-64MB blocks is it going to be wasted rest of the 63MB or will it be allocated to other file?
Block size 64MB means an upper bound size for a block. It doesn't mean that file blocks less than 64MB will consume 64MB. It will not consume 64MB to store a chunk of 1MB.
If the file is 160 megabytes,
Hope this helps.
According to this page. Looks like it'll be one 64 MB block and one 1 MB block.
HDFS is often blissfully unaware that the final record in one block may be only a partial record, with the rest of its content shunted off to the following block. HDFS only wants to make sure that files are split into evenly sized blocks that match the predefined block size for the Hadoop instance... Not every file you need to store is an exact multiple of your system’s block size, so the final data block for a file uses only as much space as is needed.
The answer is 2 blocks, one 64MB and other 1MB.
HDFS just like other filesystems splits the file into blocks and then saves those blocks to disks.
But there are two major differences between them :
HDFS block sizes are huge because every block has a metadata record at namenode, smaller block sizes means a lot of blocks and overloading of namenode with metadata.
Hence, bigger block sizes used in HDFS.
HDFS block sizes are just an abstraction on the linux based file system, hence 65MB will use one 64MB block and other 1MB space from second block, rest 63MB from second block is still free and available for other data.
That is, Namenode will have two blocks recorded for 65MB but the actual file system space is 65MB only.
I have taken below Quoting from Hadoop - The Definitive Guide:
Note, however, that small files do not take up any more disk space than is required to store the raw contents of the file. For example, a 1 MB file stored with a block size of 128 MB uses 1 MB of disk space, not 128 MB,
Here my questions
1) 1 MB file stored with a block size of 128 MB uses 1 MB of disk space, not 128 MB.) How does hdfs use the remaining 127M in this block?
2)Is there any chance to store another file in same block?
1 MB file stored in 128MB block with 3 replication. Then the file will be stored in 3 blocks and uses 3*1=3 MB only not 3*128=384 MB. But it shows each the block size as 128 MB. It is just an abstraction to store the metadata in the namenode, but not an actual memory size used.
No way to store more than a file in a single block. Each file will be stored in a separate block.
Reference:
https://stackoverflow.com/a/21274388/3496666
https://stackoverflow.com/a/15065274/3496666
https://stackoverflow.com/a/14109147/3496666
NameNode Memory Usage:
Every file, directory and block in HDFS is represented as an object. i.e. each entry i the namenode is reflected to a item.
in the namenode’s memory, and each of object/item occupies 150 to 200 bytes of namenode memory.memorandums prefer fewer large files as a result of the metadata that needs to be stored.
Consider a 1 GB file with the default block size of 64MB.
-Stored as a single file 1 GB file
Name: 1 item
Block=16
Total Item = 16*3( Replication factor=3) = 48 + 1(filename) = 49
Total NameNode memory: 150*49
-Stored as 1000 individual 1 MB files
Name: 1000
Block=1000
Total Item = 1000*3( Replication factor=3) = 3000 + 1000(filename) = 4000
Total NameNode memory: 150*4000
Above results clarify that large number of small files is a overhead of naemnode memory as it takes more space of NameNode memory.
Block Name and Block ID is a unique ID of a particular block of data.This uniue ID is getting used to identified
the block during reading of the data when client make a request to read data.Hence it can not be shared.
HDFS is designed to handle large files. Lets say you have a 1000Mb file. With a 4k block size, you'd have to make 256,000
requests to get that file (1 request per block). In HDFS, those requests go across a network and come with a lot of overhead.
Each request has to be processed by the Name Node to figure out where that block can be found. That's a lot of traffic!
If you use 64Mb blocks, the number of requests goes down to 16, greatly reducing the cost of overhead and load on the Name Node.
To keep these things in mind hadoop recommend large block size.
HDFS block size is a logical unit of splitting a large file into small chunks. This chunks is basically called a block.
These chunks/block is used during further parallel processing of the data.i.e. MapReduce Programming or other model
to read/process of that within HDFS.
If a file is small enough to fit in this logical block then one block will get assigned for the file and it will
take disk space according to file size and Unix file system you are using.The detail about, how file gets stored in disk is available on this link.
HDFS block size Vs actual file size
As HDFS block size is a logical unit not a physical unit of the memory, so there is no waste of memory.
These link will be useful to understand the problem with small file.
Link1,
Link2
See Kumar's Answer
You could look into SequenceFiles or HAR Files depending on your use case. HAR files are analogous to the Tar command. MapReduce can act upon each HAR files with a little overhead. As for SequenceFiles, they are in a way a container of Key/Value pairs. The benefit of this is a Map task can act upon each of these pairs.
HAR Files
Sequence Files
More About Sequence Files