MLlib ALS cannot delete checkpoint RDDs Wrong FS: hdfs://[url] expected: file:/// - hadoop

Question

I am using Spark MLlib's ALS class to train a MatrixFactorizationModel. I have setup a HDFS for checkpointing intermediate rdds (as suggested by ALS class). The rdds are begin saved but I get an exception when it tries to delete them again: java.lang.IllegalArgumentException: Wrong FS: hdfs://[url], expected: file:///
Here is the stack trace:
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://[url], expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:642)
at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:69)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:516)
at org.apache.hadoop.fs.ChecksumFileSystem.delete(ChecksumFileSystem.java:528)
at org.apache.spark.ml.recommendation.ALS$$anonfun$2$$anonfun$apply$mcV$sp$1.apply(ALS.scala:568)
at org.apache.spark.ml.recommendation.ALS$$anonfun$2$$anonfun$apply$mcV$sp$1.apply(ALS.scala:566)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.ml.recommendation.ALS$$anonfun$2.apply$mcV$sp(ALS.scala:566)
at org.apache.spark.ml.recommendation.ALS$$anonfun$train$1.apply$mcVI$sp(ALS.scala:602)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at org.apache.spark.ml.recommendation.ALS$.train(ALS.scala:596)
at org.apache.spark.mllib.recommendation.ALS.run(ALS.scala:219)
at adapters.ALSAdapter.run(ALSAdapter.java:59) ...
The culprit seems to be:
at org.apache.spark.ml.recommendation.ALS.scala 568
FileSystem.get(sc.hadoopConfiguration).delete(new Path(file), true)
which appears to return a RawLocalFilesystem instead of a distributed filesystem object.
I have not touched sc.hadoopConfiguration. The only interaction I've had is to call myJavaStreamingContext.checkpoint(hdfs://[url + directory]);.
Is there anything further I need to do client side to setup sc.hadoopConfiguration or would the problem be hdfs server side?
I was using spark 1.3.1 but tried 1.4.1 and the problem still persists.

Answer 1

your sc.hadoopConfiguration is using local filesystem ("fs.default.name" = file:///) and despite specifying explicitly the checkpoint location, the ALS code uses sc.hadoopConfiguration to determine filesystem (e.g. for deleting previous checkpoints if applicable et al)
you can try
set sc.hadoopConfiguration.set("fs.default.name", hfds://xx:port)
set sc.setCheckpointDir(yourCheckpointDir)
and just call checkpoint with no explicit filesystem path or see if setCheckpointInterval will work for you (i.e. let ALS take care of checkpointing)
you can add hdfs-site.xml and core-site.xml to spark classpath if you don't want sc.hadoopConfiguration.set in code