Why is TreeMap reset after every reduce method? - hadoop

Question

In my reduce method, I want to operate with the TreeMap variable reduceMap to aggregate the incoming key values. However, this map loses it's state with every reduce method call. Subsequently Hadoop prints only the very last value (plus the test values I added) that is put into the TreeMap. Why is that? It does work as I intend it in my map method.
public static class TopReducer extends
Reducer<Text, IntWritable, Text, IntWritable> {
private TreeMap<Text, Integer> reducedMap = new TreeMap<Text, Integer>();
#Override
public void reduce(Text key, Iterable<IntWritable> values,
Context context) throws IOException, InterruptedException {
int sum = 0;
String strValues = "";
for (IntWritable value : values) {
sum += value.get();
strValues += value.get() + ", ";
}
System.out.println("Map size Before: " +reducedMap);
Integer val = sum;
if (reducedMap.containsKey(key))
val += reducedMap.get(key);
// Only add, if value is of top 30.
reducedMap.put(key, val);
System.out.println("Map size After: " +reducedMap);
reducedMap.put(new Text("test"), 77777);
System.out.println("REDUCER: rcv: (" + key + "), " + "(" + sum
+ "), (" + strValues + "):: new (" + val + ")");
}
/**
* Flush top 30 context to the next phase.
*/
#Override
protected void cleanup(Context context) throws IOException,
InterruptedException {
System.out.println("-----FLUSHING TOP " + TOP_N
+ " MAPPING RESULTS-------");
System.out.println("MapSize: " + reducedMap);
int i = 0;
for (Entry<Text, Integer> entry : entriesSortedByValues(reducedMap)) {
System.out.println("key " + entry.getKey() + ", value "
+ entry.getValue());
context.write(entry.getKey(), new IntWritable(entry.getValue()));
if (i >= TOP_N)
break;
else
i++;
}
}
}

Answer 1

Hadoop re-uses object references for efficiency purposes - so when you call reducedMap.put(key, val) the key value will match a key already in the map (because Hadoop had just replaced the contents of your key object, not given you a new reference to a new object with new contents). It's effectively the same as calling the following:
Text key = new Text("x");
reducedMap.put(key, val); // map will be of size 1
key.set("y");
reducedMap.put(key, val); // map will still be of size 1
// as it will be comparing key to the itself
// and just updating the mapped value val
You need to make a deep copy of your key before putting it into the map:
reducedMap.put(new Text(key), val)