FileNotFoundException on saveAsNewHadoopFile - hadoop

Question

I'm performing bulk loading of data into HBase using Spark. My Python script works perfectly to do this job, however I need to be able to submit this using spark submit so that I can run it on the cluster.
When I run the script locally with the following:
#!/bin/bash
sudo /usr/hdp/current/spark-client/bin/spark-submit\
--master local[*]\
--deploy-mode client\
--verbose\
--num-executors 3\
--executor-cores 1\
--executor-memory 512m\
--driver-memory 512m\
--conf\
spark.logConf=true\
/test/BulkLoader.py
it works perfectly - loads the data, writes the HFiles, bulk loads them. However, when I run the code with YARN as follows:
#!/bin/bash
sudo /usr/hdp/current/spark-client/bin/spark-submit\
--master yarn\
--deploy-mode client\
--verbose\
--num-executors 3\
--executor-cores 1\
--executor-memory 512m\
--driver-memory 512m\
--conf\
spark.logConf=true\
--conf\
spark.speculation=false\
/test/BulkLoader.py
things go wrong quickly. As soon as the script tries to write the HFile, I get the following error:
An error occurred while calling z:org.apache.spark.api.python.PythonRDD.saveAsNewAPIHadoopFile.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 15.0 failed 26 times, most recent failure:
Lost task 0.25 in stage 15.0 (TID 67, sandbox.hortonworks.com): java.io.FileNotFoundException: File file:/tmp/hfiles-06-46-57/_temporary/0/_temporary/attempt_201602150647_0019_r_000000_25/f1 does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:609)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:822)
...
When writing the HFile, a FileNotFoundException is raised on a _temporary directory. I looked around and found that many other people had encountered such errors (here and here) however nothing suggested worked for me. I set the number of executors to 1 and set speculation to false since this was suggested as the likely cause of the error, however the problem persists. If anybody could suggest other options for me to explore, I'd be grateful.