Aggregated information and projection in Pig Latin - hadoop

Question

I'm trying to apply a maximum aggregate function to a table by grouping on some fields and projecting. Can I refer to other non-grouping fields in the original table in the aggregating projection?
As as example I have a table blah with schema (user_id: long, order_id: long, product_id: long, gender: chararray, size: int), where user_id, order_id and product_id create a composite key but there can be multiple user ids and order ids. To get the maximum size for each order I use
result_table = foreach (group blah by (user_id, order_id)) generate
FLATTEN(group) as (user_id, order_id),
MAX(blah.size) as max_size;
Is there some way I can also add product_id to the creation of result_table so I have a table containing the user_id, order_id, product_id and max_size (max_size would be duplicated over differing product_ids) ?
If I could refer to the product_id specific to each grouped user_id and order_id I can save myself a mapreduce job by not joining back with the original table to access this field. Thanks guys.

Answer 1

Pig is well suited for such things, it has bags and that enables it to do things which in SQL require extra joins.
If you do the following:
grp = group blah by (user_id, order_id);
describe grp;
you will see that there is a bag with the schema identical to the schema of the "blah" (something like group:(user_id:long, order_id: long), blah: {(user_id: long, order_id: long, product_id: long, gender: chararray, size: int)}). That is a very powerful thing as it will allow us to create an output with all of the original rows with group summaries in each row without using inner joins:
grp = group blah by (user_id, order_id);
result_table = foreach grp generate
FLATTEN(blah.(user_id, order_id, product_id)), -- flatten the bag created by original group
MAX(blah.size) as max_size;
if the same product_id appears multiple times within group of user_id, order_id than there will be duplicates, to avoid it we could use a DISTINCT nested into FOREACH:
grp = group blah by (user_id, order_id);
result_table = foreach grp {
dist = distinct blah.(user_id, order_id, product_id); -- remove duplicates
generate flatten(dist), MAX(blah.size) as max_size;
}
It will be done in a single MapReduce job.