Hadoop MapReduce is not producing desired output - hadoop

Question

I have one large file that contain patent information. The header is as follow "PATENT","GYEAR","GDATE","APPYEAR","COUNTRY","POSTATE","ASSIGNEE","ASSCODE","CLAIMS".
I want to calculate the average claims per patent by year, where the key is the year and value is the average amount. However, reducer output shows that my average amount is 1.0 all the time. Where did my program go wrong?
The Main class
public static void main(String [] args) throws Exception{
int res = ToolRunner.run(new Configuration(), new AvgClaimsByYear(), args);
System.exit(res);
}
The Driver class
Configuration config = this.getConf();
Job job = Job.getInstance(config, "average claims per year");
job.setJarByClass(AvgClaimsByYear.class);
job.setMapperClass(TheMapper.class);
job.setPartitionerClass(ThePartitioner.class);
job.setNumReduceTasks(4);
job.setReducerClass(TheReducer.class);
job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
return job.waitForCompletion(true) ? 0 : 1;
The Mapper class
public static class TheMapper extends Mapper<LongWritable, Text, IntWritable, IntWritable> {
private IntWritable yearAsKeyOut = new IntWritable();
private IntWritable claimsAsValueOut = new IntWritable(1);
#Override
public void map(LongWritable keyIn, Text valueIn, Context context) throws IOException,InterruptedException {
String line = valueIn.toString();
if(line.contains("PATENT")) {
return; //skip header
}
else {
String [] patentData = line.split(",");
yearAsKeyOut.set(Integer.parseInt(patentData[1]));
if (patentData[8].length() > 0) {
claimsAsValueOut.set(Integer.parseInt(patentData[8]));
}
}
context.write(yearAsKeyOut, claimsAsValueOut);
}
}
The Partitioner Class
public static class ThePartitioner extends Partitioner<IntWritable, IntWritable> {
public int getPartition(IntWritable keyIn, IntWritable valueIn, int totalNumPartition) {
int theYear = keyIn.get();
if (theYear <= 1970) {
return 0;
}
else if(theYear > 1970 && theYear <= 1979) {
return 1;
}
else if(theYear > 1979 && theYear <=1989) {
return 2;
}
else{
return 3;
}
}
}
The Reducer class
public static class TheReducer extends Reducer<IntWritable,IntWritable,IntWritable,FloatWritable> {
#Override
public void reduce(IntWritable yearKey, Iterable<IntWritable> values, Context context) throws IOException,InterruptedException {
int totalClaimsThatYear = 0;
int totalPatentCountThatYear = 0;
FloatWritable avgClaim = new FloatWritable();
for(IntWritable value : values) {
totalClaimsThatYear += value.get();
totalPatentCountThatYear += 1;
}
avgClaim.set(calculateAvgClaimPerPatent (totalPatentCountThatYear, totalClaimsThatYear));
context.write(yearKey, avgClaim);
}
public float calculateAvgClaimPerPatent (int totalPatentCount, int totalClaims) {
return (float)totalClaims/totalPatentCount;
}
}
The Input
3070801,1963,1096,,"BE","",,1,,269,6,69,,1,,0,,,,,,,
3070802,1963,1096,,"US","TX",,1,,2,6,63,,0,,,,,,,,,
3070803,1963,1096,,"US","IL",,1,,2,6,63,,9,,0.3704,,,,,,,
3070804,1963,1096,,"US","OH",,1,,2,6,63,,3,,0.6667,,,,,,,
3070805,1963,1096,,"US","CA",,1,,2,6,63,,1,,0,,,,,,,
The Output
1963 1.0
1964 1.0
1965 1.0
1966 1.0
1967 1.0
1968 1.0
1969 1.0
1970 1.0

Answer 1

In calculateAvgClaimPerPatent() your expression performs integer division before conversion to a float. Convert the two integers to float before the division.
-- edit --
Also, looking over the code again, the average written out is really the average number of claims per record, grouped by the 4 intervals defined by your partitioner. In other words, the number of claims for one patent in 1972 is being averaged in with the number of claims for a different patent in 1975. That doesn't match your problem description.