Where is script like wordcount executed in MapReduce? - hadoop

Question

I am asking for knowledge? When running a MapReduce with a wordcount jar when does the code execute? Is it during the mapper task or the driver method?

Answer 1

When running a MapReduce with a wordcount jar when does the code execute?
It executes with main i.e. Driver code and then map code followed by reducer code (if any)
Is it during the mapper task or the driver method?
Yes its both.
Driver - would drive the map reduce, where you define which class i should use for mapper, reducer, paritioner, combiner.
Mapper - mapper is where your word count would create a map of key value pair with key as word and value as number of times it occured.
Reducer - Reducer would take the value from each mapper and would sum all the value against same key accross mapper and would give you the final result.

Answer 2

as correctly answered by SMA , code execution starts with main method of driver class , which passes control to mapper and reducer class using methods setMapperClass , setReducerClass of job object.

Answer 3

Have a look at official map reduce tutorial for better understanding. I am using key points to explain the example.
Let's have a look at java word count example.
Assume that you have created wc.jar as follows.
$ jar cf wc.jar WordCount*.class
Now run the WordCount example
$ bin/hadoop jar wc.jar WordCount /user/joe/wordcount/input /user/joe/wordcount/output
Now you have passed input directory & output directory for reading and writing of data respectively.
Assuming that:
/user/joe/wordcount/input - input directory in HDFS
/user/joe/wordcount/output - output directory in HDFS
input directory in HDFS is used by Mapper to read the data & output directory in HDFS is used by reducer to store data.
Applications typically implement the Mapper and Reducer interfaces to provide the map and reduce methods. These form the core of the job.
If you see the main method of WordCount class (you can call it as Driver program),
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
We are setting Mapper class, Reducer class, Combiner class, Output Key & Value classes and Input/Output file locations.
Some jobs may have only Mapper. Some jobs will have Mapper and Reducer. Some jobs will have Partitioner & Combiner in addition to Mapper & Reducer classes. So basically the flow is decided by you and Hadoop framework will form a workflow depending on your inputs.
For above example :
Mapper will read input data from HDFS file location from FileInputFormat.addInputPath API. Mapper is set by below line
job.setMapperClass(TokenizerMapper.class);
Combiner, which is a mini reducer will run on Mapper output. It will reduce network IO by combining output of Mappers.
job.setCombinerClass(IntSumReducer.class);
Reducer is set by below API.
job.setReducerClass(IntSumReducer.class);
Mapper
Mapper maps input key/value pairs to a set of intermediate key/value pairs. The Hadoop MapReduce framework spawns one map task for each InputSplit generated by the InputFormat for the job.
Reducer
Reducer reduces a set of intermediate values which share a key to a smaller set of values.
Combiner:
Users can optionally specify a combiner, via Job.setCombinerClass(Class), to perform local aggregation of the intermediate outputs, which helps to cut down the amount of data transferred from the Mapper to the Reducer.