PIG:Twitter Sentiment Analysis - hadoop

Question

I am trying to implement the twitter sentiment analysis.I need to get all positive tweets and negative tweets and store them in particular text files.
sample.json
{"id": 252479809098223616, "created_at": "Wed Apr 12 08:23:20 +0000 2016", "text": "google is a good company", "user_id": 450990391}{"id": 252479809098223616, "created_at": "Wed Apr 12 08:23:20 +0000 2016", "text": "facebook is a bad company","user_id": 450990391}
dictionary.text having all the positive and negetive words list
weaksubj 1 bad adj n negative
strongsubj 1 good adj n positive
Pig Script:-
tweets = load 'new.json' using JsonLoader('id:chararray,text:chararray,user_id:chararray,created_at:chararray');
dictionary = load 'dictionary.text' AS (type:chararray,length:chararray,word:chararray,pos:chararray,stemmed:chararray,polarity:chararray);
words = foreach tweets generate FLATTEN( TOKENIZE(text) ) AS word,id,text,user_id,created_at;
sentiment = join words by word left outer, dictionary by word;
senti2 = foreach sentiment generate words::id as id,words::created_at as created_at,words::text as text,words::user_id as user_id,dictionary::polarity as polarity;
res = FILTER senti2 BY polarity MATCHES '.*possitive.*';
describe res:-
res: {id: chararray,created_at: chararray,text: chararray,user_id: chararray,polarity: chararray}
But when I dump res I dont see any output, but it executes fine without any errors.
What is the mistake that I am doing here.
Please suggest me.
Mohan.V

Answer 1

I see 2 errors here
1 : line 2 - When you DUMP dictionary , you will see all the records
in column 1 with rest of columns showing empty.
Solution : Specify an appropriate delimiter using PigStorage();
dictionary = load 'dictionary.text' AS (type:chararray,length:chararray,word:chararray,pos:chararray,stemmed:chararray,polarity:chararray);
DUMP dictionary;
(weaksubj 1 bad adj n negative,,,,,)
(strongsubj 1 good adj n positive,,,,,)
Second error :
line 6 : Correct the spelling of positive ! use something like
res = FILTER senti2 BY UPPER(polarity) MATCHES '.*POSITIVE.*';

Answer 2

I see spelling mistake in:
res = FILTER senti2 BY polarity MATCHES '.*possitive.*';
Isn't it '.*positive.*' ?

Answer 3

As per my recommendations you should use custom UDF's for solving your problem . Now you can use elephant-bird-pig-4.1.jar,json-simple-1.1.1.jar .
Also if you wanted to look at example for these then you can use these Sentiment Analysis Tutorial .
If you wanted code then you can refer these code and format your code according to tutorial and my code ,
REGISTER ‘/usr/local/elephant-bird-hadoop-compat-4.1.jar';
REGISTER '/ usr/local /elephant-bird-pig-4.1.jar';
REGISTER '/ usr/local /json-simple-1.1.1.jar’;
load_tweets = LOAD '/user/new.json' USING com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad') AS myMap;
extract_details = FOREACH load_tweets GENERATE myMap#'id' as id,myMap#'text' as text;
tokens = foreach extract_details generate id,text, FLATTEN(TOKENIZE(text)) As word;
dictionary = load '/user/dictionary.text' AS (type:chararray,length:chararray,word:chararray,pos:chararray,stemmed:chararray,polarity:chararray);
word_rating = join tokens by word left outer, dictionary by word using 'replicated’; describe word_rating;
rating = foreach word_rating generate tokens::id as id,tokens::text as text, dictionary::rating as rate;
word_group = group rating by (id,text);
avg_rate = foreach word_group generate group, AVG(rating.rate) as tweet_rating;
positive_tweets = filter avg_rate by tweet_rating>=0;