First Hadoop program using map and reducer - hadoop

Question

I'm trying to compile my first Hadoop program. I have as input file something like that:
1 54875451 2015 LA89LP
2 47451451 2015 LA89LP
3 878451 2015 LA89LP
4 54875 2015 LA89LP
5 2212 2015 LA89LP
When I'm compiling it i get map 100%, reducer 0% and an java.lang.Exception: java.util.NoSuchElementException caused by a lot of staff, including:
java.util.NoSuchElementException
java.util.StringTokenizer.nextToken(StringTokenizer.java:349)
I don't really understand why. Any help is really appreciate
My Map and Reducer are in this way:
public class Draft {
public static class TokenizerMapper extends Mapper<Object, Text, Text, Text>{
private Text word = new Text();
private Text word2 = new Text();
public void map(Object key, Text value, Context context
) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
String id = itr.nextToken();
String price = itr.nextToken();
String dateTransfer = itr.nextToken();
String postcode = itr.nextToken();
word.set(postcode);
word2.set(price);
context.write(word, word2);
}
}
}
public static class MaxReducer extends Reducer<Text,Text,Text,Text> {
private Text word = new Text();
private Text word2 = new Text();
public void reduce(Text key, Iterable<Text> values, Context context
) throws IOException, InterruptedException {
String max = "0";
HashSet<String> S = new HashSet<String>();
for (Text val: values) {
String d = key.toString();
String price = val.toString();
if (S.contains(d)) {
if (Integer.parseInt(price)>Integer.parseInt(max)) max = price;
} else {
S.add(d);
max = price;
}
}
word.set(key.toString());
word2.set(max);
context.write(word, word2);
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "Draft");
job.setJarByClass(Draft.class);
job.setMapperClass(TokenizerMapper.class);
job.setReducerClass(MaxReducer.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
job.setOutputKeyClass(Text.class); // output key type for mapper
job.setOutputValueClass(Text.class); // output value type for mapper
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}

Answer 1

This error occurs, when some of your records have less than 4 fields. Your code in the mapper assumes that each record contains 4 fields: id, price, dateTransfer and postcode.
But, some of the records may not contain all the 4 fields.
For e.g. if the record is:
1 54875451 2015
then, following line will throw an exception (java.util.NoSuchElementException):
String postcode = itr.nextToken();
You are trying to assign postcode (which is assumed to be the 4th field), but there are only 3 fields in the input record.
To overcome this problem, you need to change your string tokenizer code in the map() method. Since you are emitting only postcode and price from the map(), you can change your are code as below:
String[] tokens = value.toString().split(" ");
String price = "";
String postcode = "";
if(tokens.length >= 2)
price = tokens[1];
if(tokens.length >= 4)
postcode = tokens[3];
if(!price.isEmpty())
{
word.set(postcode);
word2.set(price);
context.write(word, word2);
}