Spark is shuffling large amount of data - hadoop

Question

I have written a spark job. Which looks like below :
public class TestClass {
public static void main(String[] args){
String masterIp = args[0];
String appName = args[1];
String inputFile = args[2];
String output = args[3];
SparkConf conf = new SparkConf().setMaster(masterIp).setAppName(appName);
JavaSparkContext sparkContext = new JavaSparkContext(conf);
JavaRDD<String> rdd = sparkContext.textFile(inputFile);
Integer[] keyColumns = new Integer[] {0,1,2};
Broadcast<Integer[]> broadcastJob = sparkContext.broadcast(keyColumns);
Function<Integer,Long> createCombiner = v1 -> Long.valueOf(v1);
Function2<Long, Integer, Long> mergeValue = (v1,v2) -> v1+v2;
Function2<Long, Long, Long> mergeCombiners = (v1,v2) -> v1+v2;
JavaPairRDD<String, Long> pairRDD = rdd.mapToPair(new PairFunction<String, String, Integer>() {
private static final long serialVersionUID = -6293440291696487370L;
#Override
public Tuple2<String, Integer> call(String t) throws Exception {
String[] record = t.split(",");
Integer[] keyColumns = broadcastJob.value();
StringBuilder key = new StringBuilder();
for (int index = 0; index < keyColumns.length; index++) {
key.append(record[keyColumns[index]]);
}
key.append("|id=1");
Integer value = new Integer(record[4]);
return new Tuple2<String, Integer>(key.toString(),value);
}}).combineByKey(createCombiner, mergeValue, mergeCombiners).reduceByKey((v1,v2) -> v1+v2);
pairRDD.saveAsTextFile(output);
}
}
The program calculates the sum of values for each key.
As per my understanding, the local combiner should run on each node and add up the values for same keys and
then shuffling occurs with little amount of data.
But on SparkUI it is showing huge amount of shuffle read and shuffle write(almost 58GB).
Am I doing anything wrong?
How to know if the local combiner is working?
Cluster Details :-
20 Nodes cluster
Each Node having 80GB HardDisk, 8GB RAM, 4 cores
Hadoop-2.7.2
Spark-2.0.2(prebuild-with-Hadoop-2.7.x distribution)
Input file details :-
input file is stored on hdfs
input file size : 400GB
number of records : 16,129,999,990
record columns : String(2 char),int,int,String(2 char),int,int,String(2 char),String(2 char),String(2 char)
Note :
Max Number of distinct keys is 1081600.
In spark logs I see the task running with localitylevel NODE_LOCAL.

Answer 1

Let's decompose this problem and see what get. To simplify computations lets assume that:
Total number of records is 1.6e8
Number of unique keys is 1e6
Split size is 128MB (this seems to be consistent with the number of task in you UI).
With these values data will be spitted into ~3200 partitions (3125 in your case). This gives you around 51200 records per split. Furthermore if distribution of number of values per key is uniform there should ~160 records per key on average.
If data is randomly distributed (it is not sorted by key for example) you can expect that on average number of records per key per partition will be close to one*. This is basically the worst case scenario where map side combine doesn't reduce amount of data at all.
Furthermore you have to remember that size of a flat file typically will be significant lower that size of the serialized objects.
With real life data you can typically expect some type of order emerging from data collection process so things should be better than what we calculated above but the bottom line is that, if data is not already grouped by partition, map side combine may provide no improvements at all.
You could probably decrease amount of shuffled data by using a bit larger split (256MB would give you a bit over 100K per partition) but it comes at price of longer GC pauses and possibly other GC issues.
* You can either simulate this by taking samples with replacement:
import pandas as pd
import numpy as np
(pd
.DataFrame({"x": np.random.choice(np.arange(3200), size=160, replace=True)})
.groupby("x")
.x.count()
.mean())
or just think about the problem of randomly assigning 160 balls to 3200 buckets.