Hadoop GroupingComparator class purpose - hadoop

Question

I'm implementing a join between two datasets A and B by a String key, which is the name attribute. I need to match similar names in this join.
My first thought, given that I was implementing secondary sort to get the values extracted from database A before the values from database B, was to create a grouping comparator class and instead of using the compareTo method to group values by the natural key, use a string similarity algorithm, but it has not worked as expected. See my code below.
public class StringSimilarityGroupingComparator extends WritableComparator {
protected StringSimilarityGroupingComparator() {
super(JoinKeyTagPairWritable.class, true);
}
public int compare(WritableComparable w1, WritableComparable w2) {
JoinKeyTagPairWritable k1 = (JoinKeyTagPairWritable) w1;
JoinKeyTagPairWritable k2 = (JoinKeyTagPairWritable) w2;
StringSimilarityMatcher nameMatcher = new StringSimilarityMatcher(
StringSimilarityMatcher.NAME_MATCH);
return nameMatcher.match(k1.getJoinKey(), k2.getJoinKey()) ? 0 : k1
.getJoinKey().compareTo(k2.getJoinKey());
}
This approach makes total sense to me. Where was I mistaken? Isn't this the purpose of overriding the grouping comparator class?
EDIT:
I'm aware that I need to write a custom partitioner to guarantee that similar keys are sent to the same reducer, but as I'm dealing with a relatively small database the job can run fine with only one reducer.
To clarify the problem I'm facing I ran the job with an identity reducer to expose which keys are been grouped together, I'm emitting the key and the dataset tag. Here is a sample of the output:
Ricardo 0
Ricardo 1
Ricardo 1
Ricardo Beguer 1
END OF REDUCE METHOD
Ricardo Castro 1
END OF REDUCE METHOD
Ricardo S.(Gueguel) 1
Ricardo Silva 1
END OF REDUCE METHOD
Ricardo tsubasa 1
Ricardo! 1
RicardoRoale 1
END OF REDUCE METHOD
All these names are matching using my algorithm, but they are not been grouped together. I'm not understanding why this is happening, since I don't know how MapReduce uses my grouping comparator class to group keys.
The dataset tagged with 0 is the left database of the the join, hence, I expect all the similar names from dataset 1 to be grouped with a name from dataset 0.
Can you define how MapReduce does this grouping? Is it after sort and iteratively?
I've seen many people talk about set-similarity (e.g. this paper) when dealing with the problem of matching similar names, but this approach seems simpler and also efficient, since names are not large strings and the matching is done by the grouping comparator class and only one job is needed.
Thanks in advance!

Answer 1

You didn't describe the way your solution wasn't working correctly, but from what you've shown I can make a few suggestions.
The first issue I see is that you don't guarantee that similar names get sent to the same reducer. For example, I'd hope that "Chris" and "Christopher" would compare as being the same in your name matcher, but you don't guarantee that keys "Chris" get sent to the same reducer as keys "Christopher". If you use the default partitioner then it's quite possible that "Chris" with hashcode 65087095 gets assigned a different reducer than does "Christopher" with hashcode 1731528407.
I suggest, for correctness and performance both, that you try to normalize each name in the mapper, so where your mappers might write:
"Christopher" -> value
you'd instead have them write:
"Chris" -> "Christopher" + original value
Where "Chris" is the normalized form of all similar names ("Chris", "Christopher", "Christophe", etc.). In this way, the default partitioners and groupers would work correctly and you'd get the grouping you want with the passed key/value data you want.
You may also be dealing with a more difficult issue and that is that a name like "Chris" might actually be similar to two names that aren't themselves similar, like "Christopher" and "Christine". If this is the case then things get really gnarly. A solution is still possible, but you may need more information (e.g. gender) to normalize the name or you may have to accept missed matches. I can elaborate if this is the situation you've got.
--EDIT--
To address your clarification... There are two sorters used on the key/value pairs before they're passed to the reducer. The first sorts the keys and if no grouper is specified then the reducer is called according to unique key values. If a grouper is specified then the grouper is only used to compare "adjacent keys" (per the first sort) to see if they should be passed into the same reducer call.
For example, say A1 and A2 are to be considered the same key (e.g. similar names) but B is not similar to A1 or A2. If the sorter were to sort the keys as
A1 A2 B
then the reducer would be called twice, once for A1 and A2 and again for B. If, however, the sort managed to produce the key sequence:
A1 B A2
then the reducer would be called three times. The grouper comparator only compares A1 and B and then B and A2.
This means that the grouper sort really should compare strings the same way as the sorter does, but only with more matches.
To use your example above, the grouper seems to be comparing "Ricardo Beguer" and "Ricardo Castro" and finding them not similar. Even though "Ricardo" might be similar to "Ricardo Castro", those two are never compared.
Can you test all names against each other to see if any pair is not similar?

Answer 2

I think Chris is right. The main rule that you likely are violating is
If A < B (via sort) then A <= B (via grouper)
If A = B (via sort) then A = B (via grouper)