Custom object as value for Mapper output - hadoop

Question

I have object have constructed as following:
Class ObjExample {
String s;
Object[] objArray; // element in this array can be primitive type or array of primitive type.
}
I know that to using it as output type for mapper or reducer, we have to implement WritableComparable for it.
But I really get confused how to write readFields(), write(), compareTo() for this kind of class?

Answer 1

You can wrap field s in Text and objArray in ArrayWritable. Each element of the objArray would be an array (also ArrayWritable) of primitives. Here is possible implementation:
public static final class ObjExample implements WritableComparable<ObjExample> {
public final Text s = new Text(); // wrapped String
public final ArrayOfArrays objArray = new ArrayOfArrays();
#Override
public int compareTo(ObjExample o) {
// your logic here, example:
return s.compareTo(o.s);
}
#Override
public void write(DataOutput dataOutput) throws IOException {
s.write(dataOutput);
objArray.write(dataOutput);
}
#Override
public void readFields(DataInput dataInput) throws IOException {
s.readFields(dataInput);
objArray.readFields(dataInput);
}
// set size of the objArray
public void setSize(int n) {
objArray.set(new IntArray[n]);
}
// set i-th element of the objArray to an array of elements
public void setElement(int i, IntWritable... elements) {
IntArray subArr = new IntArray();
subArr.set(elements);
objArray.get()[i] = subArr;
}
}
You will need two more classes to make it work:
// array of primitives
public static final class IntArray extends ArrayWritable {
public IntArray() {
// you can specify any other primitive wrapper (DoubleWritable, Text, ...)
super(IntWritable.class);
}
}
// array of arrays
public static final class ArrayOfArrays extends ArrayWritable {
public ArrayOfArrays() {
super(IntArray.class);
}
}
Example of construction of the object:
ObjExample o = new ObjExample();
o.s.set("hello");
o.setSize(2);
o.setElement(0, new IntWritable(0)); // single primitive
o.setElement(1, new IntWritable(1), new IntWritable(2)); // array of primitives