Pig: Read multiple files and append column-wise - hadoop

I want to achieve this in Pig but not sure about an efficient way.
I have an input file(with header: COL1,COL2,COL3,COL4,TAG) and multiple "value" files all with similar format (TAG,VALUE). I want to append the "VALUE" column of each "value" file with the input file based on "TAG" as key column. So if there are 3 "value" files then format of final combined file will be (COL1,COL2,COL3,COL4,TAG,VALUE1,VALUE2,VALUE3).
One approach I can think of is to read each "value" file and then join with input file in an incremental way. So we will have multiple intermediate files.
Like first join input file with one value file and output will be : COL1,COL2,COL3,COL4,TAG,VALUE1 .
Now this becomes new input file and join with another "value" file and output will be COL1,COL2,COL3,COL4,TAG,VALUE1,VALUE2.
Is there a better way ?

You could use COGROUP with multiple relations, it will cause only one MR job. Following code was typed without testing, but the idea should work:
header = LOAD 'header_path' using PigStorage(',') AS (COL1,COL2,COL3,COL4,TAG);
tv_1 = LOAD 'tv_1' using PigStorage(',') AS (TAG,VALUE);
tv_2 = LOAD 'tv_2' using PigStorage(',') AS (TAG,VALUE);
tv_3 = LOAD 'tv_3' using PigStorage(',') AS (TAG,VALUE);
joined = COGROUP header BY TAG, tv_1 BY TAG, tv_2 BY TAG, tv_3 BY TAG;
result = FOREACH joined GENERATE FLATTEN(header), FLATTEN((IsEmpty(tv_1) ? TOBAG(TOTUPLE(null) : tv_1.VALUE)) AS VALUE1, FLATTEN((IsEmpty(tv_2) ? TOBAG(TOTUPLE(null) : tv_2.VALUE)) AS VALUE2, FLATTEN((IsEmpty(tv_3) ? TOBAG(TOTUPLE(null) : tv_3.VALUE)) AS VALUE3;

Related

How to load data in Pig from .tsv file?

I have a .TSV file containing data in HDFS and i am not able to load it into Pig.
The command i am using is "
A = load 'file_location' as (name:chararray, age:int, gpa:float);
B = foreach A generate (name, age);
DUMP B;
Error returned : Unable to find operator for alias A
If you do not specify the delimiter PIG uses default ',' as the delimiter for loading the file.Hence your load statement is failing.You have to explicitly specify the delimiter '\t'.
A = LOAD 'file_location' USING PigStorage('\t') AS (name:chararray, age:int, gpa:float);
Do it like this
A = load 'path/of/file' using PigStorage('\t') AS (name:chararray,age:int,gpa:float);
B = foreach A generate name, age;
DUMP B;
ps: I don't think there is any fault with your commands.As tab (\t) is default delimiter for pig . I am getting the correct output with your commands .Can you please send me logs or screenshot of your terminal.

PIG: Process tuples in a bag

I have a data set that looks like this after a GROUP operation :
input = key1|{(a1,b1,c1),(a2,b2,c2)}
key2|{(a3,b3,c3),(a4,b4,c4),(a5,b5,c5)}
I need to traverse the above to generate final output like this :
<KEY>key1</KEY>|
<VALUES><VALUE><VALUE1>a1</VALUE1>VALUE2>b1</VALUE2>VALUE3>c1</VALUE3></VALUE><VALUE><VALUE1>a2</VALUE1><VALUE2>b2</VALUE2><VALUE3>c2</VALUE3> </VALUE></VALUES>
<KEY>key2</KEY>| ...
I have tried to use FLATTEN and CONCAT to achieve this result in the below manner:
A = FOREACH input GENERATE key, FLATTEN(input);
output = FOREACH A GENERATE CONCAT('<KEY>',CONCAT(input.key,'</KEY>')),
CONCAT('<VALUE>',''),
CONCAT('<VALUE1>',CONCAT(input.col1,'</VALUE1>')
...
But this does not give the desired output. Fairly new to pig, so don't know if this is possible.
If you FLATTEN your bag than you'll ended up as many new 'rows' as many elements you had in the bag:
key1|(a1,b1,c1)
key1|(a2,b2,c2)
If I understand your problem correctly you want this:
Use the BagToTuple built in function.
Than you'll get
key1|(a1,b1,c1,a2,b2,c2)
After this you can format your data with e.g. a UDF

how to join header row to detail rows in multiple files with apache pig

I have several CSV files in a HDFS folder which I load to a relation with:
source = LOAD '$data' USING PigStorage(','); --the $data is a passed as a parameter to the pig command.
When I dump it, the structure of the source relation is as follows: (note that the data is text qualified but I will deal with that using the REPLACE function)
("HEADER","20110118","20101218","20110118","T00002")
("0000000000000000035412","20110107","2699","D","20110107","2315.","","","","","","C")
("0000000000000000035412","20110107","2699","D","20110107","246..","162","74","","","","B")
<.... more records ....>
("HEADER","20110224","20110109","20110224","T00002")
("0000000000000000035412","20110121","2028","D","20110121","a6c3.","","","","","R","P")
("0000000000000000035412","20110217","2619","D","20110217","a6c3.","","","","","R","P")
<.... more records ....>
So each file has a header which provides some information about the data set that follows it such as the provider of the data and the date range it covers.
So now, how can I transform the above structure and create a new relation like the following ?:
{
(HEADER,20110118,20101218,20110118,T00002),{(0000000000000000035412,20110107,2699,D,20110107,2315.,,,,,,C),(0000000000000000035412,20110107,2699,D,20110107,246..,162,74,,,,B),..more tuples..},
(HEADER,20110224,20110109,20110224,T00002),{(0000000000000000035412,20110121,2028,D,20110121,a6c3.,,,,,R,P),(0000000000000000035412,20110217,2619,D,20110217,a6c3.,,,,,R,P),..more tuples..},..more tuples..
}
Where each header tuple is followed by a bag of record tuples belonging to that header ?.
Unfortunately there is no common key field between the header and the detail rows, so I don't think cant use any JOIN operation. ?
I am quite new to Pig and Hadoop and this is one of the first concept projects that I am engaging in.
Hope my question is clear and look forward to some guidance here.
This should get you started.
Code:
Source = LOAD '$data' USING PigStorage(',','-tagFile');
A = SPLIT Source INTO FileHeaders IF $1 == 'HEADER', FileData OTHERWISE;
B = GROUP FileData BY $0;
C = GROUP FileHeaders BY $0;
D = JOIN B BY Group, C BY Group;
...

Store pig result in a text file

Hi stackoverflow community;
i'm totally new to pig, i want to STORE the result in a text file and name it as i want. is it possible do this using STORE function.
My code:
a = LOAD 'example.csv' USING PigStorage(';');
b = FOREACH a GENERATE $0,$1,$2,$3,$6,$7,$8,$9,$11,$12,$13,$14,$20,$24,$25;
STORE b INTO ‘myoutput’;
Thanks.
Yes you will be able to store your result in myoutput.txt and you can load the data into file with any delimiter you want using PigStorage.
a = LOAD 'example.csv' USING PigStorage(';');
b = FOREACH a GENERATE $0,$1,$2,$3,$6,$7,$8,$9,$11,$12,$13,$14,$20,$24,$25;
STORE b INTO ‘myoutput.txt’ using PigStorage(';');
Yes, it is possible. b will store every row into 25 different columns - $0 to S25.

Pig reading data as databytearray

Hey guys i have one more question I am just not able to understand the behavior of pig
I am loading the data into pig and after some transformation storing it using PigStorage() on hdfs(/user/sga/transformeddata).
But when I load the data from /user/sga/transformeddata location and do
temp = load '/user/sga/transformeddata' using PigStorage();
gen = foreach temp generate page_type;
dump gen;
getting following error:
databytearray can not be cast to java.lang.String
but if i do
gen = foreach temp generate *;
dump gen;
it works fine
any help is totally appreciated to understand this.
As required presenting the code:
STORE union_of_all_records INTO '/staged/google/data_after_denormalization' using PigStorage('\t','-schema');
union_of_all_records is an alias in pig.
now another script which will consume this data
lookup_data =
LOAD '/staged/google/page_type_map_file/' using PigStorage() AS (page_type:chararray,page_type_classification:chararray);
load_denorm_clickstream_record =
LOAD '/staged/google/data_after_denormalization' using PigStorage('\t','-schema');
and join on these two aliases
denorm_clickstream_record = LIMIT load_denorm_clickstream_record 100;
join_with_lookup =
JOIN denorm_clickstream_record BY page_type LEFT OUTER, lookup_data BY page_type;
step x : final_output =
FOREACH join_with_lookup
GENERATE denorm_clickstream_record::page_type as page_type;
at step x i get the above error.
I think you have to options:
1) You have to tell Pig the schema that the data has. For example:
temp = load '/user/sga/transformeddata' using PigStorage() AS (page_type:chararray);
2) When you first store the data tell Pigstorage to store the schema information as well. PigStorage('\t', '-schema'); When you load the data as you do above, PigStorage should read the schema from the schema information.

Resources