Spark custom serializer leads to ClassNotFound - hadoop

Question

I'm trying to use custom spark serializer defined as:
conf.set("spark.serializer", CustomSparkSerializer.class.getCanonicalName());
But when I submit application to Spark I'm facing issue with ClassNotFoundException when executor env creating, for example:
16/04/01 18:41:11 INFO util.Utils: Successfully started service 'sparkExecutor' on port 52153.
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1643)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:68)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:149)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:250)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: java.lang.ClassNotFoundException: example.CustomSparkSerializer
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at **org.apache.spark.util.Utils$.classForName(Utils.scala:173)**
at org.apache.spark.SparkEnv$.instantiateClass$1(SparkEnv.scala:266)
at org.apache.spark.SparkEnv$.instantiateClassFromConf$1(SparkEnv.scala:287)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:290)
at org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:218)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:183)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:69)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:68)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
In local standalone mode it can be solved using "spark.executor.extraClassPath=path/to/jar", but on cluster with several nodes it does not help.
I have tried all known (for me) approaches as such as use --jars, executor (and even driver) extra class and library path, sc.addJar also... It was not help.
I found that Spark uses specific classloader in org.apache.spark.util.Utils$.classForName(Utils.scala:173) for load serializer class, but I really don't understand how to make custom serializer loadable.
The application flow submit is more complex - Oozie -> SparkSubmit -> YARN client -> Spark application
The question is - does anybody know how to use custom spark serializer and how to resolve ClassNotFound issue with it ?
Thanks in advance!

Answer 1

The reason why it happens is because I use spark.executor.extraClassPath with prefix /home/some_user. It seems that Spark can not load any class from that path because Spark process owner is another user, once I putted JAR to path smth like /usr/lib/ everything works fine.
So, I confused with users and Hadoop/Oozie/Spark processes owners, but I was not expecting such behavior from ClassLoaders =)
Thank you for help!