How to tune SparkPageRank example application for shorter garbage collection? - performance

Question

I am running a Spark application in a 7 node cluster - 1 driver and 6 executors on amazon EC2 machines. I use 6 m4.2xlarge instances with 1 executor each. They have 8 cores each. The driver is on a m4.xlarge VM, which had 4 cores. The spark version is 2.1.1.
I use the following command to start SparkPageRank application.
spark-submit \
--name "ABC" \
--master spark://xxx:7077 \
--conf spark.driver.memory=10g \
--conf "spark.app.name=ABC" \
--conf "spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:ConcGCThreads=5" \
--class org.apache.spark.examples.SparkPageRank \
--executor-memory 22g \
/home/ubuntu/spark-2.1.1/examples/target/scala-2.11/jars/spark-examples_2.11-2.1.1.jar /hdfscheck/pagerank_data_11G_repl1.txt 4
The GC time using these configuration comes out to be really high.
Here is a chunk of the GC log for one of the executor:
1810.053: [GC pause (GCLocker Initiated GC) (young), 0.1694102 secs]
[Parallel Time: 167.8 ms, GC Workers: 8]
[GC Worker Start (ms): Min: 1810053.2, Avg: 1810053.3, Max: 1810053.4, Diff: 0.1]
[Ext Root Scanning (ms): Min: 0.2, Avg: 0.4, Max: 0.7, Diff: 0.5, Sum: 2.9]
[Update RS (ms): Min: 12.4, Avg: 12.7, Max: 13.2, Diff: 0.7, Sum: 101.4]
[Processed Buffers: Min: 11, Avg: 12.9, Max: 16, Diff: 5, Sum: 103]
[Scan RS (ms): Min: 29.4, Avg: 29.8, Max: 30.1, Diff: 0.7, Sum: 238.7]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Object Copy (ms): Min: 124.5, Avg: 124.6, Max: 124.7, Diff: 0.1, Sum: 996.9]
[Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
[Termination Attempts: Min: 1, Avg: 2.2, Max: 5, Diff: 4, Sum: 18]
[GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
[GC Worker Total (ms): Min: 167.5, Avg: 167.5, Max: 167.6, Diff: 0.1, Sum: 1340.2]
[GC Worker End (ms): Min: 1810220.8, Avg: 1810220.8, Max: 1810220.8, Diff: 0.0]
[Code Root Fixup: 0.0 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.4 ms]
[Other: 1.2 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 0.5 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.4 ms]
[Humongous Register: 0.0 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 0.1 ms]
[Eden: 992.0M(960.0M)->0.0B(960.0M) Survivors: 160.0M->160.0M Heap: 14.6G(22.0G)->13.8G(22.0G)]
[Times: user=1.34 sys=0.00, real=0.17 secs]
(More at https://pastebin.com/E5bbQZgD)
I could only see one fishy thing that the concurrent-mark-end took a lot of time.
I would appreciate if someone could tell me how to tune garbage collection for this particular case. The VM on which driver node is located has 16GB of memory, whereas the executor VMs have 32GB memory.

Answer 1

(Not really an answer but just a collection of hints to help out).
My driver node has 16GB of memory
I don't think it's the case given you executed spark-submit with spark.driver.memory=10g.
You should use --driver-memory instead (which is just a shortcut, but makes things a little easier to remember):
--driver-memory MEM Memory for driver (e.g. 1000M, 2G) (Default: 1024M).
Regarding your main question, the high use of GC that seems to be how the PageRank algorithm works. Note the high use of Shuffle Write with Input not that large.
I also think that the GC time is not that long as compared to Task Time.
I'm concerned with RDD Blocks being only 2 as that seems to suggest the parallelism being very low, but that might be how it should work.
run-example SparkPageRank
The following can be replaced with a simple run-example SparkPageRank (as described in Spark's Where to Go from Here)
spark-submit ... \
--class org.apache.spark.examples.SparkPageRank ... \
/home/ubuntu/spark-2.1.1/examples/target/scala-2.11/jars/spark-examples_2.11-2.1.1.jar