I am newbie on Hadoop. I remember I learned from somewhere that in Hadoop, all map functions have to be completed before reduce functions can start off.
But I just got the printout when I run a map reduce program like this:
map(15%), reduce(5%)
map(20%), reduce(7%)
map(30%), reduce(10%)
map(38%), reduce(17%)
map(40%), reduce(25%)
why they run in parallel?
Before actual Reduce phase starts, Shuffle, Sort and Merge take place as Mappers keep on completing. This percentage signifies that. It is not the actual Reduce phase. This happens in parallel to reduce the overhead which would otherwise be incurred if framework keeps on waiting for completion of all the Mappers first and then do the Shuffling, Sorting and Merging.
Related
Does the shuffle and sort phase come before the end of the map task or does it come after the output is generated from the map task so that there is no look back to the map task anymore. This is a 'Map only task' case where I get confusion.
If there is no Shuffle and sort in Map only task, can someone explain how is the data written into the final output files.
When you have a map-only task, there is not shuffling at all, which means that mappers will write the final output directly to the HDFS.
On the other hand, when you have a whole Map-Reduce program, with mappers and reducers, yes, shuffling can start before reduce-phase start.
Quoting this very nice answer in SO:
First of all shuffling is the process of transfering data from the
mappers to the reducers, so I think it is obvious that it is necessary
for the reducers, since otherwise, they wouldn't be able to have any
input (or input from every mapper). Shuffling can start even before
the map phase has finished, to save some time. That's why you can see
a reduce status greater than 0% (but less than 33%) when the map
status is not yet 100%.
Hope this answer had clarified your confusion.
I see that whenever I run a Map Reduce task the hadoop job shows me the percentage of Map and Reduce tasks done.
I understand that both mappers and reducers run in a distributed fashion and can report how much they have processed to the controller.
But how does the controller know total data to be processed ? If the controller tries to figure out size of all the input files I would image that will be inefficient. Is it some kind of crude approximation ?
I didn't read all the code related to this part in hadoop. but some thought about it, hope it helps
map task completion percentage computation: when a slave machine complete a map task , it will let the master machine know, so it is convenient for master and controller to know the percentage of map task competition, but if the process is blocked or killed the task completed by this process will be reassigned , and the map completed value will decrease in the case.
reduce task completion percentage computation: reduce phrase include shuffle task and reduce task, shuffle task start when the map task completion reach 5% , shuffle task mainly copy the data from the map task output directory. so the data copied seems to be domain factor in shuffle part, since the merge sort can be run at the same time. in reduce task, the completion percentage can easily compute by how may "iterators" has been processed.
ps: as vehtfym metioned in the comment: Reduce phase includes copy(finishing at 33%), sort(finishing at 66%) and reduce (finishing at 100%)
I am a beginner to hadoop & when I am running a hadoop job I noticed the progress log which shows map 80% reduce 25%. My understanding of map reduce is that mappers produce bunch of intermediate values. After mappers producing output there is shuffle/sort of intermediate pairs & these values are sent to reduce job. Can someone please explain me how map/reduce can work in parallel.
The outputs from the mappers have to be copied to the appropriate reducer nodes. This is called the shuffle process. That can start even before all the mappers have finished, since the decision of which key goes to which reducer is dependent only on the output key from the mapper. So the 25% progress you see is due to the shuffle phase.
After shuffle, there is a sort phase and then the reduce phase. Sort and reduce cannot happen unless all mappers have completed. Since shuffle can happen before the mappers finish, you can see a maximum of 33.33% reduce completion before the mappers have finished. This is because the default apache implementation considers shuffle, sort and reduce each to take an equal 33.33% of the time.
I have a question.. I have a mapreduce program that get input from cassandra. my input is a little big, about 100000000 data. my problem is that my program takes too long to process, but I think mapreduce is good and fast for large volume of data. so I think maybe I have problems in number of map and reduce tasks.. I set the number of map and reduce asks with JobConf, with Job, and also in conf/mapred-site.xml, but I don't see any changes.. in my logs at first there is map 0% reduce 0% and after about 2 hours working it shows map 1% reduce 0%..!!
what should I do? please Help me I really get confused...
Please consider these points to check where the bottleneck might be --
Merely configuring to increase the number of map or reduce tasks files won't do. You need hardware to support that. Hadoop is fast, but to process a huge file, as you have mentioned
you need to have more numbers of parellel map and reduce tasks
running. To achieve what you need more processors. To get more
processors you need more machines (nodes). For example, if you have
2 machines with 8 processors each, you get a total processing power
of around 16. So, total 16 map and reduce tasks can run in parallel and the next set of tasks comes in as soon as slots gets unoccupied out of the 16 slots you have.
Now, when you add one more machine with 8 processors, you now have 24.
The Algorithms you used for map and reduce. Even though, you have
processing power, that doesn't mean your Hadoop application will
perform unless your algorithm performs. It might be the case that
a single map task takes forever to complete.
I have a Hadoop job running which has finished the map part for for 4 days, and now it seems to get suspended at the stage of reduce (reducer is done for 30%)
I really hope to have a way to only re-process the reduce part if at all possible without having to re-process the long-run map part, Any suggestions?
Something probably gets it worse that I only have one reducer.
Hadoop will only restart the Reduce step in your case.
However, if your job fails, you can't just skip the map step.
In this case you should probably split the two stages into seperate jobs, especially if your mapper is computational intensive.