Perform count on distinct values of a bag in Pig - hadoop

Question

I have a question on Pig when performing what seems like two levels of groupings. As an example, let's say I had some example input data like:
email_id:chararray from:chararray to:bag{recipients:tuple(recipient:chararray)}
e1 user1#example.com {(friend1#example.com),(friend2#example.com),(friend3#myusers.com)}
e2 user1#example.com {(friend1#example.com),(friend4#example.com)}
e3 user1#example.com {(friend5#example.com)}
e4 user2#example.com {(friend2#example.com),(friend4#example.com)}
So each line is an email from a user "from" to user(s) "to".
And I ultimately want a list of all senders and all the people they've sent emails to, including the # of emails sent for each person, sorted from highest to lowest, for example:
user1#example.com {(friend1#example.com, 2), (friend2#example.com, 1), (friend3#example.com, 1), (friend4#example.com, 1), (friend5#example.com, 1)}
user2#example.com {(friend2#example.com, 1), (friend4#example.com, 1)}
Ideas on the best way to tackle this in Pig would be appreciated!

Answer 1

Here is one version of the script:
inpt = load '/pig_data/pig_fun/input/from_senders.txt' as (email_id:chararray, from:chararray, to:bag{recipients:tuple(recipient:chararray)});
pivot = foreach inpt generate from, FLATTEN(to);
pivot = foreach pivot generate from, to::recipient as recipient;
dump pivot;
/*
(user1#example.com,friend1#example.com)
(user1#example.com,friend2#example.com)
(user1#example.com,friend3#myusers.com)
(user1#example.com,friend1#example.com)
(user1#example.com,friend4#example.com)
(user1#example.com,friend5#example.com)
(user2#example.com,friend2#example.com)
(user2#example.com,friend4#example.com)
*/
grp = group pivot by (from, recipient);
with_count = foreach grp generate FLATTEN(group), COUNT(pivot) as count;
dump with_count;
/*
(user1#example.com,friend1#example.com,2)
(user1#example.com,friend2#example.com,1)
(user1#example.com,friend3#myusers.com,1)
(user1#example.com,friend4#example.com,1)
(user1#example.com,friend5#example.com,1)
(user2#example.com,friend2#example.com,1)
(user2#example.com,friend4#example.com,1)
*/
to_bag = group with_count by from;
result = foreach to_bag {
order_by_count = order with_count by count desc;
generate group as from, order_by_count.(recipient, count);
};
dump result;
/*
(user1#example.com,{(friend1#example.com,2),(friend2#example.com,1),(friend3#myusers.com,1),(friend4#example.com,1),(friend5#example.com,1)})
(user2#example.com,{(friend2#example.com,1),(friend4#example.com,1)})
*/
Hope it helps.