Pig reduce job stuck at 50% while running group command - hadoop

Question

I have loaded a file with some 6000 data lines using the following commands
A = load '/home/hduser/hdfsdrive/piginput/data/airlines.dat' using PigStorage(',') as (Airline_ID:int, Name:chararray, Alias:chararray, IATA:chararray, ICAO:chararray, Callsign:chararray, Country:chararray, Active:chararray);
B = foreach airline generate Country,Airline_ID;
C = group B by Country;
D = foreach C generate group,COUNT(B);
In the above code, I could execute the first 3 commands without any issues, but 4th one is running for a long time. I tried the following
dump C;
Even this stuck at the same place. Here is the log:
2016-04-20 16:08:16,617 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library 2016-04-20 16:08:16,898 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
already exists! 2016-04-20 16:08:17,125 INFO
org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2016-04-20 16:08:17,129 INFO org.apache.hadoop.mapred.Task: Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin#1da9647b
2016-04-20 16:08:17,190 INFO org.apache.hadoop.mapred.ReduceTask:
ShuffleRamManager: MemoryLimit=130652568,
MaxSingleShuffleLimit=32663142 2016-04-20 16:08:17,195 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Thread started: Thread for
merging on-disk files 2016-04-20 16:08:17,195 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Thread started: Thread for
merging in memory files 2016-04-20 16:08:17,195 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Thread waiting: Thread for
merging on-disk files 2016-04-20 16:08:17,196 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Need another 1 map output(s)
where 0 is already in progress 2016-04-20 16:08:17,196 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Thread started: Thread for
polling Map Completion Events 2016-04-20 16:08:17,196 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Scheduled 0 outputs (0 slow hosts
and0 dup hosts) 2016-04-20 16:08:22,197 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Scheduled 1 outputs (0 slow hosts
and0 dup hosts) 2016-04-20 16:09:18,202 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Need another 1 map output(s)
where 1 is already in progress 2016-04-20 16:09:18,203 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Scheduled 0 outputs (0 slow hosts
and0 dup hosts) 2016-04-20 16:10:18,208 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Need another 1 map output(s)
where 1 is already in progress 2016-04-20 16:10:18,208 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Scheduled 0 outputs (0 slow hosts
and0 dup hosts) 2016-04-20 16:11:18,214 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Need another 1 map output(s)
where 1 is already in progress 2016-04-20 16:11:18,214 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Scheduled 0 outputs (0 slow hosts
and0 dup hosts) 2016-04-20 16:11:22,395 WARN
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 copy failed:
attempt_201604201138_0003_m_000000_0 from ubuntu 2016-04-20
16:11:22,396 WARN org.apache.hadoop.mapred.ReduceTask:
java.net.SocketTimeoutException: connect timed out at
java.net.PlainSocketImpl.socketConnect(Native Method) at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
java.net.Socket.connect(Socket.java:579) at
sun.net.NetworkClient.doConnect(NetworkClient.java:175) at
sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at
sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at
sun.net.www.http.HttpClient.(HttpClient.java:211) at
sun.net.www.http.HttpClient.New(HttpClient.java:308) at
sun.net.www.http.HttpClient.New(HttpClient.java:326) at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:998)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:934)
at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:852)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(ReduceTask.java:1636)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.setupSecureConnection(ReduceTask.java:1593)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1493)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1401)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1333)
2016-04-20 16:11:22,398 INFO org.apache.hadoop.mapred.ReduceTask: Task
attempt_201604201138_0003_r_000000_0: Failed fetch #1 from
attempt_201604201138_0003_m_000000_0 2016-04-20 16:11:22,398 WARN
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 adding host ubuntu to penalty
box, next contact in 12 seconds 2016-04-20 16:11:22,398 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0: Got 1 map-outputs from previous
failures 2016-04-20 16:11:37,399 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Scheduled 1 outputs (0 slow hosts
and0 dup hosts) 2016-04-20 16:12:19,403 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Need another 1 map output(s)
where 1 is already in progress 2016-04-20 16:12:19,403 INFO
org.apache.hadoop.mapred.ReduceTask:
attempt_201604201138_0003_r_000000_0 Scheduled 0 outputs (0 slow hosts
and0 dup hosts)
Even I stopped all the jobs and tried restarted, but no use. My environment is Ubuntu/ Hadoop 1.2.1/ Pig 0.15.0
Please help.
Thanks, Sathish

Answer 1

I resolved this. Issue is with the IP address configured in /etc/hosts was incorrect. I updated this to the IP address assigned to the Ubuntu machine and restart the Hadoop services. I found this mismatch from hadoop-hduser-jobtracker-ubuntu.log, where it says:
STARTUP_MSG: host = ubuntu/10.1.0.249
And in hadoop-hduser-datanode-ubuntu.log, it throw following error:
2016-04-25 12:23:05,738 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 10.1.6.173/10.1.6.173:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
Based on these errors, I could track the issue with IP address and fixed it in /etc/hosts file, restarted the server. After this, all the Hadoop jobs are running without any issues and I could run load the data and run PIG scripts.
Thanks, Sathish.

Answer 2

You are loading the data to relation A but generating B from relation airline?
B = foreach airline generate Country,Airline_ID;
Should be
B = foreach A generate Country,Airline_ID;
Also if you are counting the number of Airlines per country, you will have to modify the relation D as
D = foreach C generate group as Country,COUNT(B.Airline_ID);