Let's say the key-value pairs with the keys “the”, “sound”, “is” are processed by reducer 1 and the key-value pairs with the keys “it”, “right”, “sounds” are processed by reducer 2.
What would be the outputs of the two reducers?
Would the output file of each reducer be sorted then combined then sorted again?
When the reducer receives them is it already sorted alphabetically so that reducer 1 receives “is”, “it”, “right” and reducer 2 receives “the”, “sound”, “sounds”?
To answer your queries:
Output of the reducer would be the word and count of its occurrence.
The output of reducer working on different keys are never combined. There is no such phase in mapreduce.
The output of the mapper is sorted and fed into reducer; but different reducer emits its output randomly and the output of the all the reducers is Not sorted again. There is no such phase in mapreduce.
Even though reducers are getting keys in sorted order, think each reducer running into a separate JVM and a separate process. They output the data without "knowing" that there are more reducer running.
Related
As per my understanding, mapper runs first followed by partitioner(if any) followed by Reducer. But if we use Partitioner class, I am not sure when Sorting and Shuffling phase runs?
A CLOSER LOOK
Below diagram explain the complete details.
From this diagram, you can see where the mapper and reducer components of the Word Count application fit in, and how it achieves its objective. We will now examine this system in a bit closer detail.
mapreduce-flow
Shuffle and Sort phase will always execute(across the mappers and reducers nodes).
The hierarchy of the different phase in MapReduce as below:
Map --> Partition --> Combiner(optional) --> Shuffle and Sort --> Reduce.
The short answer is: Data sorting runs on the reducers, shuffling/sorting runs before the reducer (always) and after the map/combiner(if any)/partitioner(if any).
The long answer is that into a MapReduce job there are 4 main players:
Mapper, Combiner, Partitioner, Reducer. All these are classes you can actually implement by your own.
Let's take the famous word count program, and let's assume the split where we are working contains:
pippo, pluto, pippo, pippo, paperino, pluto, paperone, paperino, paperino
and each word is record.
Mapper
Each mapper runs over a subset of your file, its task is to read each record from the split and assign a key to each record which will output.
Mapper will store intermediate result on disk ( local disk ).
The intermediate output from this stage will be
pippo,1
pluto,1
pippo,1
pippo,1
peperino,1
pluto,1
paperone,1
paperino,1
paperino,1
At this will be stored on the local disk of the node which runs the mapper.
Combiner
It's a mini-reducer and can aggregate data. It can also run joins, so called map-join. This object helps to save bandwidth into the cluster because it aggregates data on the local node.
The output from the combiner, which is still part of the mapper phase, will be:
pippo,3
pluto,2
paperino,3
paperone,1
Of course here are the data from ONE node. Now we have to send the data to the reducers in order to get the global result. Which reducer will process the record depends on the partitioner.
Partitioner
It's task is to spread the data across all the available reducers. This object will read the output from the combiner and will select the reducer which will process the key.
In this example we have two reducers and we use the following rule:
all the pippo goes to reducer 1
all the pluto goes to reducer 2
all the paperino goes to reducer 2
all the paperone goes to reducer 1
so all the nodes will send records which have the key pippo to the same reducer(1), all the nodes will send the records which have the key pluto to the same reducer (2) and so on...
Here is where the data get shuffled/sorted and, since the combiner already reduced the data locally, this node has to send only 4 record instead of 9.
Reducer
This object is able to aggregate the data from each node and it's also able to sort the data.
I would like to move only the first 10 records of the output after sort/shuffle to the reducer. Is this possible?
The reason is this: I am to find the least 10 items with the largest count in a file. However, I know that the results of the mapping phase will be arrive at the reducer already sorted. Hence, instead of sorting in the mappers, I'd like to just pass only the first 10 lines after 'shuffle and sort' to the reducer. this will allow the reducer sort only a subset of the original record.
Is there any way to do this?
You can achieve this by writing a custom Combiner for the job.
The different stages in the MapReduce job are:
Mapper -> Partitioner -> Sorting -> Combiner -> Reducer.
Now Combiner logic only read the first 10 (n) records and discord all the other. The Reducer will receive only 10 records from each Mapper/Combiner.
Comment provided by #K246:
From haodop definitive guide (4th ed) : Before it writes to disk, the thread first divides the data into partitions corresponding to the reducers that they will ultimately be sent to. Within each partition, the background thread performs an in-memory sort by key, and if there is a combiner function, it is run on the output of the sort.
When you say least 10 in the file...Is it for each mapper or for the entire input.
If for each mapper, then you have to aggregate the again at reducer from all mappers. Then as #YoungHobbit pointed out, Combiner will do the work.
If you need least 10 from entire input file, then I think, you need to handle it with a single reducer and output accordingly.
Also, you said in the last line, that reducer will sort only subset. Do you mean you are sorting again in Reducer or that some logic is performed in reducer for only subset of the input.
I have few basic questions in Hadoop MapReduce.
Assume if 100 mappers were executed and zero reducer. Will it
generate 100 files? All individual are sorted? Across all mapper
output are sorted?
Input for reducer is Key -> Values. For each key, all values are sorted?
Assume if 50 reducers were executed. Will it generate 50 files? All individual files are sorted? Across all reducer output are sorted?
Is there any place where guaranteed sorting happens in MapReduce?
1.Assume if 100 mappers were executed and zero reducer. Will it generate 100 files?
Yes.
All individual are sorted?
No. If no reducers are used, then the output of mappers are not sorted. Sorting only takes place when there is a reduce phase.
Across all mapper output are sorted?
No, for the same reason, as above.
2.Input for reducer is Key -> Values. For each key, all values are sorted?
No. However, the keys are sorted. After the shuffling phase, in which the reducer gets the output of the mappers, it merge-sorts the sorted output keys of the mappers (since there IS a reduce phase) and when it starts reducing, the keys are sorted.
3.Assume if 50 reducers were executed. Will it generate 50 files?
Yes. (unless you use MultipleOutputs)
All individual files are sorted?
No. The sorted input does not guarantee a sorted output. The output depends on the algorithm that you use in the reduce method.
Across all reducer output are sorted?
No, for the same reason as above. However, if you use an Identity Reducer, i.e., you just write the input of the reducer as you get it, the reducer's output will be sorted PER REDUCER, not globally.
Is there any place where guaranteed sorting happens in MapReduce?
Sorting takes place when there is a reduce phase and it is applied in the output keys of each mapper and the input keys of each reducer. If you want to globally sort the input of the reducer, you can either use a single reducer, or a TotalOrderPartitioner, which is a bit tricky...
I am a bit confused with the output I get from Mapper.
For example, when I run a simple wordcount program, with this input text:
hello world
Hadoop programming
mapreduce wordcount
lets see if this works
12345678
hello world
mapreduce wordcount
this is the output that I get:
12345678 1
Hadoop 1
hello 1
hello 1
if 1
lets 1
mapreduce 1
mapreduce 1
programming 1
see 1
this 1
wordcount 1
wordcount 1
works 1
world 1
world 1
As you can see, the output from mapper is already sorted. I did not run Reducer at all.
But I find in a different project that the output from mapper is not sorted.
So I am totally clear about this..
My questions are:
Is the mapper's output always sorted?
Is the sort phase integrated into the mapper phase already, so that the output of map phase is already sorted in the intermediate data?
Is there a way to collect the data from sort and shuffle phase and persist it before it goes to Reducer? A reducer is presented with a key and a list of iterables. Is there a way, I could persist this data?
Is the mapper's output always sorted?
No. It is not sorted if you use no reducer. If you use a reducer, there is a pre-sorting process before the mapper's output is written to disk. Data gets sorted in the Reduce phase. What is happening here (just a guess) is that you are not specifying a Reducer class, which, in the new API, is translated into using the Identity Reducer (see this answer and comment). The Identity Reducer just outputs its input. To verify that, see the default Reducer counters (there should be some reduce tasks, reduce input records & groups, reduce output records...)
Is the sort phase integrated into the mapper phase already, so that the output of map phase is already sorted in the intermediate data?
As I explained in the previous question, if you use no reducers, mapper does not sort the data. If you do use reducers, the data start getting sorted from the map phase and then get merge-sorted in the reduce phase.
Is there a way to collect the data from sort and shuffle phase and persist it before it goes to Reducer. A reducer is presented with a key and a list of iterables. Is there a way, I could persist this data?
Again, shuffling and sorting are parts of the Reduce phase. An Identity Reducer will do what you want. If you want to output one key-value pair per reducer, with the values being a concatenation of the iterables, just store the iterables in memory (e.g. in a StringBuffer) and then output this concatenation as a value. If you want the map output to go straight to the program's output, without going through a reduce phase, then set in the driver class the number of reduce tasks to zero, like that:
job.setNumReduceTasks(0);
This will not get your output sorted, though. It will skip the pre-sorting process of the mapper and write the output directly to HDFS.
Point 1: output from mapper is always sorted but based on Key.
i.e. if Map method is doing this: context.write(outKey, outValue); then result will be sorted based on outKey.
Following would be some explanations to your questions
Heading ##Does the output from mapper is always sorted?
Already answered by #SurJanSR
Heading ##Does the sort phase integrated with mapper phase already, so that the output of map phase is already sorted in the intermediate data?
In a Mapreduce Job, as you know, Mapper runs on individual splits of data and across nodes where data is persisting. The result of Mapper is written TEMPORARILY before it is written to the next phase.
In the case of a reduce operation, the TEMPORARILY stored Mapper output is sorted, shuffle based on the partitioner needs before moved to the reduce operation
In the case of Map Only Job, as in your case, The temorarily stored Mapper output is sorted based on the key and written to the final output folder (as specified in your arguments for the Job).
Heading ##Is there a way to collect the data from sort and shuffle phase and persist it before it goes to Reducer. A reducer is presented with a key and a list of iterables. Is there a way, I could persist this data?
Not sure what your requirement is. using a IdentityReducer would just persist the output. I'm not sure if this answers your question.
I support the answer of vefthym.
Usually the Mapper output is sorted before storing it locally on the node. But when you are explicitely setting up numReduceTasks to 0 in the job configuration then the mapper o/p will not be sorted and written directly to HDFS.
So we cannot say that Mapper output is always sorted!
1. Is the mapper's output always sorted?
2.Is the sort phase integrated into the mapper phase already, so that the output of map phase is already sorted in the intermediate data?
From Apache MapReduceTutorial:
( Under Mapper Section )
All intermediate values associated with a given output key are subsequently grouped by the framework, and passed to the Reducer(s) to determine the final output.
The Mapper outputs are sorted and then partitioned per Reducer. The total number of partitions is the same as the number of reduce tasks for the job
( Under Reducer Section )
Reducer NONE
It is legal to set the number of reduce-tasks to zero if no reduction is desired.
In this case the outputs of the map-tasks go directly to the FileSystem, into the output path set by FileOutputFormat.setOutputPath(Job, Path). The framework does not sort the map-outputs before writing them out to the FileSystem.
3. Is there a way to collect the data from sort and shuffle phase and persist it before it goes to Reducer? A reducer is presented with a key and a list of iterables. Is there a way, I could persist this data?
I don't think so. From Apache condemnation on Reducer:
Reducer has 3 primary phases:
Shuffle:
The Reducer copies the sorted output from each Mapper using HTTP across the network.
Sort:
The framework merge sorts Reducer inputs by keys (since different Mappers may have output the same key).
The shuffle and sort phases occur simultaneously i.e. while outputs are being fetched they are merged.
Reduce:
The output of the reduce task is typically written to a RecordWriter via TaskInputOutputContext.write(Object, Object).
The output of the Reducer is not re-sorted.
As per the documentation, the shuffle and sort phase is driven by framework
If you want to persist the data, set number of reducers to Zero, which causes persistence of Map output into HDFS but it won't sort the data.
Have a look at related SE question:
hadoop: difference between 0 reducer and identity reducer?
I did not find IdentityReducer in Hadoop 2.x version:
identityreducer in the new Hadoop API
Just got start with hadoop, got several questions about execution of reducer.
When the key, value pairs distributed to one reducer task. Does it process sequential or parallel.
For example (A,5) (A,3) (B,10) for the reducer task. Does A,B get into reducer in parallel?
When one reducer is used, the KV pairs are not processed in parallel, but are processed in sorted order. In your example above, the pairs will be sent from one or more mapper tasks (in parallel if multiple mappers) to the single reduce task. Before these values are passed to your reducer class, they are aggregated ((A,5) and (A,3) are turn into (A,{5,3})) and then sorted before the reducer task actually runs user code to 'reduce' the input sets.