Spark Streaming checkpoint to amazon s3 - spark-streaming

Question

I am trying to checkpoint the rdd to non-hdfs system. From DSE document it seems that it is not possible to use cassandra file system. So I am planning to use amazon s3 . But I am not able to find any good example to use the AWS.
Questions
How do I use Amazon S3 as checkpoint directory ?Is it just enough to call
ssc.checkpoint(amazons3url) ?
Is it possible to have any other reliable data storage other than hadoop file system for checkpoint ?

Answer 1

From the answer in the link
Solution 1:
export AWS_ACCESS_KEY_ID=<your access>
export AWS_SECRET_ACCESS_KEY=<your secret>
ssc.checkpoint(checkpointDirectory)
Set the checkpoint directory as S3 URL -
s3n://spark-streaming/checkpoint
And then launch your spark application using spark submit.
This works in spark 1.4.2
solution 2:
val hadoopConf: Configuration = new Configuration()
hadoopConf.set("fs.s3.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
hadoopConf.set("fs.s3n.awsAccessKeyId", "id-1")
hadoopConf.set("fs.s3n.awsSecretAccessKey", "secret-key")
StreamingContext.getOrCreate(checkPointDir, () => {
createStreamingContext(checkPointDir, config)
}, hadoopConf)

Answer 2

To Checkpoint to S3, you can pass the following notation to StreamingContext def checkpoint(directory: String): Unit method
s3n://<aws-access-key>:<aws-secret-key>#<s3-bucket>/<prefix ...>
Another reliable file system not listed in the Spark Documentation for checkpointing, is Taychyon