How can I using Pig scripts to generate nested Avro field? - hadoop

Question

I am new to Pig, My input data is in the format as
Record1:
{
label:int,
id: long
},
Record 2:
{
...
}
...
And what I want as output is to get
Record 1:
{
data:{
label:int,
id:long
}
},
Record 2:
{
...
}
...
I tried:
result = FOREACH input GENERATE (id, label) AS data;
but this results in a nested tuple structure that looks as below:
Record 1:
{
data:{
TUPLE_1:{
label:int,
id: long
}
}
}
How could I get rid of the one more bag as "TUPLE_1", that looks like I missed a trivial setting.

Answer 1

You probably need to specify schema when you STORE the data.
If you use org.apache.pig.piggybank.storage.avro.AvroStorage, it can take schema definition as parameter.
result = FOREACH input GENERATE label:int, id:long;
STORE result INTO 'result.avro' USING org.apache.pig.piggybank.storage.avro.AvroStorage('schema', '{"type": "record","name": "data","fields": [{"name": "label","type": "int"},{"name": "id", "type": "long"}]}');

Answer 2

My final solution is like this:
First of all to create an avro file with a certain schema, I make sure I do the following schema configuration for AvroStorage:
STORE sth INTO 'someplace' USING AvroStorage('schema','
{
### AN AVRO SCHEMA JSON STRING ###
}
');
I found using such indentation really helps to make the schema definition cleaner. And also I need to make sure I escape all special characters, especially for quotes (they may exist in "doc", that's tricky).
Then to make sth has the correct pig structure to be stored, I need to construct the entire data structure properly. One good trick could be using DESCRIBE if there already some examples of the target data files. In my previous question, the code should look like this:
in = LOAD '$INPUT_PATHS' USING AvroStorage();
in = FOREACH in GENERATE foo.label AS label, bar.id AS id;
out = FOREACH in GENERATE TOMAP('id', (long)id, 'label', (chararray)label) AS data;
RMF $OUTPUT_PATH;
STORE out INTO '$OUTPUT_PATH USING AvroStorage('schema',
{
"type": "records",
"name": "XXItem",
"namespace": "com.xxx.xxx",
"fields": [
{
"name": "data",
"type": {"type": "map", "values" : ["string", "long", "int"]}
}
]
}
');