Adding Parallel option is failing the job in Pig - hadoop

Question

I am new to Pig and am trying to run the following PigScript on our 5-node Hadoop cluster.
The following script gives me the set intersection of two columns in a relation
register '/home/workspace/Pig/setIntersecUdf.jar';
define Inter com.cs.pig.SetIntersection();
a = load '/home/pig/pig-0.12.0/input/location.txt' as (location:chararray);
b = load '/home/pig/pig-0.12.0/input/location.txt' as (location:chararray);
c = CROSS a,b parallel 10;
c = DISTINCT c;
d = Foreach c generate $0,$1,Inter($0,$1) as intersection;
e = Filter d by intersection !='[]' parallel 10;
e = Filter e by $0!=$1 parallel 10;
Store e into '/home/Documents/Pig_output';
When I run this script without giving the 'PARALLEL' option, it runs fine. But, when I add this option, I get the following error in the tasktracker stack of the reducers and the job fails.
Java.io.IOException: org.apache.hadoop.ipc.RemoteException:
java.io.IOException: File /tmp/temp-10926921/tmp-1823693600/_temporary/_attempt_201401171541_0001_r_000000_0/part-r-00000 could only be replicated to 0 nodes, instead of 1 at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639) at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:396) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:469) at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:432) at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:404) at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:256) at
org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176) at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:650) at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418) at
org.apache.hadoop.mapred.Child$4.run(Child.java:255) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:396) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /tmp/temp-10926921/tmp-1823693600/_temporary/_attempt_201401171541_0001_r_000000_0/part-r-00000 could only be replicated to 0 nodes, instead of 1 at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639) at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at
sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:396) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at
org.apache.hadoop.ipc.Client.call(Client.java:1107) at
org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at
$Proxy2.addBlock(Unknown Source) at
sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at $Proxy2.addBlock(Unknown Source) at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686) at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546) at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749) at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
Also, the diagnostic info for the failed job comes as:
# of failed Reduce Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201401171541_0001_r_000002
Kindly, help!

Answer 1

The problem appears to be that you are running out of disk space on your reducer. The CROSS operation is very expensive, in that it writes out every possible pair of records. You have 321,372 records, so your data will blow up by a factor of 321,372 when you use CROSS.
Since your data is about 6 MB, this translates to needing to write about 1.9 TB. When you specify PARALLEL 10, this gets spread across 10 reducers, which presumably doesn't max out any single machine's disk space. But without PARALLEL 10, Pig is defaulting to a single reducer (probably because of the small size of your input) and trying to write all of the data on that single machine.
Perhaps you could aggressively FILTER your results immediately after the CROSS so that Pig will throw most of those records away before trying to write to disk. Or you'll need to re-think your data flow so that you don't need to use CROSS.