Reducer that groups by two values - hadoop

Question

I have a case in which Mapper emits data that belongs to a subgroup and the subgroup belongs to a group.
I need to add up all the values in the subgroup and find the minimal value between all subgroups of the group, for each of the groups.
So, I have an output from Mapper that looks like this
Group 1
group,subgroupId,value
Group1,1,2
Group1,1,3
Group1,1,4
Group1,2,1
Group1,2,2
Group1,3,1
Group1,3,2
Group1,3,5
Group 2
group,subgroupId,value
Group2,4,2
Group2,4,3
Group2,4,4
Group2,5,1
Group2,5,2
Group2,6,1
Group2,6,2
And my output should be
Group1, 1, (2+3+4)
Group1, 2, (1+2)
Group1, 3, (1+2+5)
Group1 min = min((2+3+4),(1+2),(1+2+5))
Same for Group 2.
So I practically need to group twice, first group by GROUP and then inside of it group by SUBGROUPID.
So I should emit the minimal sum from a group, in the given example my reducer should emit (2,3), since the minimal sum is 3 and it comes from element with id 2.
So, it seems that it could be solved best using reduce twice, first reduce would get elements grouped by id and that would be passed to the second Reducer grouped by Group id.
Does this make sense and how to implement it? I've seen ChainedMapper and ChainedReducer, but they don't fit for this purpose.
Thanks

Answer 1

If all data can fit in the memory of one machine, you can simply do all this in a single job, using a single reducer (job.setNumReducers(1);) and two temp variables. The output is emitted in the cleanup phase of the reducer. Here is the pseudocode for that, if you use the new Hadoop API (that supports the cleanup() method):
int tempKey;
int tempMin;
setup() {
tempMin = Integer.MAX_VALUE;
}
reduce(key, values) {
int sum = 0;
while (values.hasNext()) {
sum += values.next();
}
if (sum < tempMin) {
tempMin = sum;
tempKey = key;
}
}
cleanup() { //only in the new API
emit(tempKey, tempMin);
}

Answer 2

Your approach (summarized below), is how I would do it.
Job 1:
Mapper: Assigns an id to a subgroupid
Combiner/Reducer(same class): Finds the minimum value for
subgroupid.
Job 2:
Mapper: Assigns a groupid to a subgroupid.
Combiner/Reducer(same class): Finds the minimum value for
groupid.
This is best implemented in two jobs for the following reasons:
Simplifies the mapper and reducer significantly (you don't need to worry about finding all the groupids the first time around). Finding the (groupid, subgroupid) pairs in the mapper could be non-trivial. Writing the two mappers should be trivial.
Follows the map reduce programming guidelines given by Tom White in Hadoop: The Definitive Guide (Chapter 6).
An Oozie workflow can easily and simply accommodate the dependent jobs.
The intermediate file products (key:subgroupid, value: min value for subgroupid) should be small, limiting the use of network resources.