Pig Mapreduce to count letters in a row - hadoop

Question

Instead of counting words I need to count letters.
But I have problems implementing this using Apache Pig version 0.8.1-cdh3u1
Given the following input:
989;850;abcccc
29;395;aabbcc
The ouput should be:
989;850;a;1
989;850;b;1
989;850;c;4
29;395;a;2
29;395;b;2
29;395;c;2
Here is what I tried:
A = LOAD 'input' using PigStorage(';') as (x:int, y:int, content:chararray);
B = foreach A generate x, y, FLATTEN(STRSPLIT(content, '(?<=.)(?=.)', 6)) as letters;
C = foreach B generate x, y, FLATTEN(TOBAG(*)) as letters;
D = foreach C generate x, y, letters.letters as letter;
E = GROUP D BY (x,y,letter);
F = foreach E generate group.x as x, group.y as y, group.letter as letter, COUNT(D.letter) as count;
A, B and C can be dumped, but "dump D" results in "ERROR 2997: Unable to recreate exception from backed error: java.lang.ClassCastException: java.lang.Integer cannot be cast to org.apache.pig.data.Tuple"
dump C displays(despite the third value being a weird tuple):
(989,850,a)
(989,850,b)
(989,850,c)
(989,850,c)
(989,850,c)
(989,850,c)
(29,395,a)
(29,395,a)
(29,395,b)
(29,395,b)
(29,395,c)
(29,395,c)
Here are the schemas:
grunt> describe A; describe B; describe C; describe D; describe E; describe F;
A: {x: int,y: int,content: chararray}
B: {x: int,y: int,letters: bytearray}
C: {x: int,y: int,letters: (x: int,y: int,letters: bytearray)}
D: {x: int,y: int,letter: bytearray}
E: {group: (x: int,y: int,letter: bytearray),D: {x: int,y: int,letter: bytearray}}
F: {x: int,y: int,letter: bytearray,count: long}
This pig version doesn't seem to support TOBAG($2..$8), hence the TOBAG(*) which also includes x and y, but that could be sorted out synactically later...
I'd like to avoid writing a UDF, otherwise I'd simply use the Java API directly.
But I don't really get the cast error. Can someone please explain it.

Answer 1

I'd propose writing a custom UDF instead. A quick, raw implementation would look like this:
package com.example;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import org.apache.pig.EvalFunc;
import org.apache.pig.data.BagFactory;
import org.apache.pig.data.DataBag;
import org.apache.pig.data.DataType;
import org.apache.pig.data.Tuple;
import org.apache.pig.data.TupleFactory;
import org.apache.pig.impl.logicalLayer.schema.Schema;
public class CharacterCount extends EvalFunc<DataBag> {
private static final BagFactory bagFactory = BagFactory.getInstance();
private static final TupleFactory tupleFactory = TupleFactory.getInstance();
#Override
public DataBag exec(Tuple input) throws IOException {
try {
Map<Character, Integer> charMap = new HashMap<Character, Integer>();
DataBag result = bagFactory.newDefaultBag();
int x = (Integer) input.get(0);
int y = (Integer) input.get(1);
String content = (String) input.get(2);
for (int i = 0; i < content.length(); i++){
char c = content.charAt(i);
Integer count = charMap.get(c);
count = (count == null) ? 1 : count + 1;
charMap.put(c, count);
}
for (Map.Entry<Character, Integer> entry : charMap.entrySet()) {
Tuple res = tupleFactory.newTuple(4);
res.set(0, x);
res.set(1, y);
res.set(2, String.valueOf(entry.getKey()));
res.set(3, entry.getValue());
result.add(res);
}
return result;
} catch (Exception e) {
throw new RuntimeException("CharacterCount error", e);
}
}
}
Pack it in a jar and execute it:
register '/home/user/test/myjar.jar';
A = LOAD '/user/hadoop/store/sample/charcount.txt' using PigStorage(';')
as (x:int, y:int, content:chararray);
B = foreach A generate flatten(com.example.CharacterCount(x,y,content))
as (x:int, y:int, letter:chararray, count:int);
dump B;
(989,850,b,1)
(989,850,c,4)
(989,850,a,1)
(29,395,b,2)
(29,395,c,2)
(29,395,a,2)

Answer 2

I do not have 0.8 version, but could you try this one:
A = LOAD 'input' using PigStorage(';') as (x:int, y:int, content:chararray);
B = foreach A generate x, y, FLATTEN(STRSPLIT(content, '(?<=.)(?=.)', 6));
C = foreach B generate $0 as x, $1 as y, FLATTEN(TOBAG(*)) as letter;
E = GROUP C BY (x,y,letter);
F = foreach E generate group.x as x, group.y as y, group.letter as letter, COUNT(C.letter) as count;

Answer 3

You can try this
grunt> a = load 'inputfile.txt' using PigStorage(';') as (c1:chararray, c2:chararray, c3:chararray);
grunt> b = foreach a generate c1,c2,FLATTEN(TOKENIZE(REPLACE(c3,'','^'),'^')) as split_char;
grunt> c = group b by (c1,c2,split_char);
grunt> d = foreach c generate group, COUNT(b);
grunt> dump d;
Output looks like this:
((29,395,a),2)
((29,395,b),2)
((29,395,c),2)
((989,850,a),1)
((989,850,b),1)
((989,850,c),4)