How to convert DStream[(Array[String], Long)] to dataframe in Spark Streaming - spark-streaming

Question

i am trying to convert my dstream to Dataframe.here is the code that am using to convert my dstream to Dataframe
val ssc = new StreamingContext(spark.sparkContext, Seconds(10))
val kafkaParams = Map[String, Object](
"bootstrap.servers" -> "ffff.dl.uk.fff.com:8002",
"security.protocol" -> "SASL_PLAINTEXT",
"key.deserializer" -> classOf[StringDeserializer],
"value.deserializer" -> classOf[StringDeserializer],
"group.id" -> "1",
"auto.offset.reset" -> "latest",
"enable.auto.commit" -> (false: java.lang.Boolean)
)
val topics = Array("mytopic")
val from_kafkastream = KafkaUtils.createDirectStream[String,
String](
ssc,
PreferConsistent,
Subscribe[String, String](topics, kafkaParams)
)
val strmk = from_kafkastream.map(record =>
(record.value,record.timestamp))
val splitup2 = strmk.map{ case (line1, line2) =>
(line1.split(","),line2)}
case class Record(name: String, trQ: String, traW: String,traNS:
String, traned: String, tranS: String,transwer: String, trABN:
String,kafkatime: Long)
object SQLContextSingleton {
#transient private var instance: SQLContext = _
def getInstance(sparkContext: SparkContext): SQLContext = {
if (instance == null) {
instance = new SQLContext(sparkContext)
}
instance
}
}
splitup2.foreachRDD((rdd) => {
val sqlContext = SQLContextSingleton.getInstance(rdd.sparkContext)
spark.sparkContext.setLogLevel("ERROR")
import sqlContext.implicits._
val requestsDataFrame = rdd.map(w => Record(w(0).toString,
w(1).toString, w(2).toString,w(3).toString, w(4).toString,
w(5).toString,w(6).toString, w(7).toString,w(8).toString)).toDF()
// am getting issue here
requestsDataFrame.show()
})
ssc.start()
I am getting error saying following
can someone help how to convert my dstreams to DF as i am new spark world

Answer 1

Maybe the mistake is when build the Record object because , you don't pass the kafkatime , only string values, and also is tuple you can't access to the atribute array of this form.
You can try this :
import session.sqlContext.implicits._
val requestsDataFrame = rdd.map(w => Record(
w._1(0).toString,
w._1(1).toString, w._1(2).toString, w._1(3).toString, w._1.toString,
w._1(5).toString, w._1(6).toString, w._1(7).toString, w._2))
requestsDataFrame.toDF()