Hadoop - Join with MultipleInputs probably skips Reducer - hadoop

Question

So, I want to perform a reduce side join with MR. (No Hive or anything, I'm experimenting on vanilla Hadoop atm).
I have 2 input files, first goes like this:
12 13
12 15
12 16
12 23
the second is simply 12 1000.
So I assign each file to a separate mapper which actually tags each key value pair with 0 or 1 depending on its source file. And that works well. How I can tell?
I get the MapOutput as expected:
| key | |value|
12 0 1000
12 1 13
12 1 15
12 1 16 etc
My Partitioner partitions based on first part of key (ie 12).
The Reducer should join by key. Yet, the job seems to skip the reduce step.
I wonder if there's something wrong with my Driver?
My code (Hadoop v0.22, but same results with 0.20.2 with extra libs from the trunk):
Mappers
public static class JoinDegreeListMapper extends
Mapper<Text, Text, TextPair, Text> {
public void map(Text node, Text degree, Context context)
throws IOException, InterruptedException {
context.write(new TextPair(node.toString(), "0"), degree);
}
}
public static class JoinEdgeListMapper extends
Mapper<Text, Text, TextPair, Text> {
public void map(Text firstNode, Text secondNode, Context context)
throws IOException, InterruptedException {
context.write(new TextPair(firstNode.toString(), "1"), secondNode);
}
}
Reducer
public static class JoinOnFirstReducer extends
Reducer<TextPair, Text, Text, Text> {
public void reduce(TextPair key, Iterator<Text> values, Context context)
throws IOException, InterruptedException {
context.progress();
Text nodeDegree = new Text(values.next());
while (values.hasNext()) {
Text secondNode = values.next();
Text outValue = new Text(nodeDegree.toString() + "\t"
+ secondNode.toString());
context.write(key.getFirst(), outValue);
}
}
}
Partitioner
public static class JoinOnFirstPartitioner extends
Partitioner<TextPair, Text> {
#Override
public int getPartition(TextPair key, Text Value, int numOfPartitions) {
return (key.getFirst().hashCode() & Integer.MAX_VALUE) % numOfPartitions;
}
}
Driver
public int run(String[] args) throws Exception {
Path edgeListPath = new Path(args[0]);
Path nodeListPath = new Path(args[1]);
Path outputPath = new Path(args[2]);
Configuration conf = getConf();
Job job = new Job(conf);
job.setJarByClass(JoinOnFirstNode.class);
job.setJobName("Tag first node with degree");
job.setPartitionerClass(JoinOnFirstPartitioner.class);
job.setGroupingComparatorClass(TextPair.FirstComparator.class);
//job.setSortComparatorClass(TextPair.FirstComparator.class);
job.setReducerClass(JoinOnFirstReducer.class);
job.setMapOutputKeyClass(TextPair.class);
job.setMapOutputValueClass(Text.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
MultipleInputs.addInputPath(job, edgeListPath, EdgeInputFormat.class,
JoinEdgeListMapper.class);
MultipleInputs.addInputPath(job, nodeListPath, EdgeInputFormat.class,
JoinDegreeListMapper.class);
FileOutputFormat.setOutputPath(job, outputPath);
return job.waitForCompletion(true) ? 0 : 1;
}

Answer 1

My reduce function had Iterator<> instead of Iterable, so the job skipped to Identity Reducer.
I can't quite believe I overlooked that. Noob error.
And the answer came from this Q/A
Using Hadoop for the First Time, MapReduce Job does not run Reduce Phase