Why does Spark Streaming not read from Kafka topic? - amazon-ec2

Question

Spark Streaming 1.6.0
Apache Kafka 10.0.1
I use Spark Streaming to read from sample topic. The code runs with no errors or exceptions but I don't get any data on the console via print() method.
I checked to see if there are messages in the topic:
./bin/kafka-console-consumer.sh \
--zookeeper ip-172-xx-xx-xxx:2181 \
--topic sample \
--from-beginning
And I am getting the messages:
message no. 1
message no. 2
message no. 3
message no. 4
message no. 5
The command to run the streaming job:
./bin/spark-submit \
--conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:MaxDirectMemorySize=512m" \
--jars /home/ubuntu/zifferlabs/target/ZifferLabs-1-jar-with-dependencies.jar \
--class "com.zifferlabs.stream.SampleStream" \
/home/ubuntu/zifferlabs/src/main/java/com/zifferlabs/stream/SampleStream.java
Here is the entire code:
import java.util.Arrays;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaPairInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import org.apache.spark.streaming.kafka.KafkaUtils;
import kafka.serializer.DefaultDecoder;
import kafka.serializer.StringDecoder;
import scala.Tuple2;
public class SampleStream {
private static void processStream() {
SparkConf conf = new SparkConf().setAppName("sampleStream")
.setMaster("local[3]")
.set("spark.serializer", "org.apache.spark.serializer.JavaSerializer")
.set("spark.driver.memory", "2g").set("spark.streaming.blockInterval", "1000")
.set("spark.driver.allowMultipleContexts", "true")
.set("spark.scheduler.mode", "FAIR");
JavaStreamingContext jsc = new JavaStreamingContext(conf, new Duration(Long.parseLong("2000")));
String[] topics = "sample".split(",");
Set<String> topicSet = new HashSet<String>(Arrays.asList(topics));
Map<String, String> props = new HashMap<String, String>();
props.put("metadata.broker.list", "ip-172-xx-xx-xxx:9092");
props.put("kafka.consumer.id", "sample_con");
props.put("group.id", "sample_group");
props.put("zookeeper.connect", "ip-172-xx-xx-xxx:2181");
props.put("zookeeper.connection.timeout.ms", "16000");
JavaPairInputDStream<String, byte[]> kafkaStream =
KafkaUtils.createDirectStream(jsc, String.class, byte[].class, StringDecoder.class,
DefaultDecoder.class, props, topicSet);
JavaDStream<String> data = kafkaStream.map(new Function<Tuple2<String,byte[]>, String>() {
public String call(Tuple2<String, byte[]> arg0) throws Exception {
System.out.println("$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ value is: " + arg0._2().toString());
return arg0._2().toString();
}
});
data.print();
System.out.println("Spark Streaming started....");
jsc.checkpoint("/home/spark/sparkChkPoint");
jsc.start();
jsc.awaitTermination();
System.out.println("Stopped Spark Streaming");
}
public static void main(String[] args) {
processStream();
}
}

Answer 1

I think you've got the code right, but the command line to execute it is incorrect.
You spark-submit the application as follows (formatting's mine + spark.executor.extraJavaOptions removed for simplicity):
./bin/spark-submit \
--jars /home/ubuntu/zifferlabs/target/ZifferLabs-1-jar-with-dependencies.jar \
--class "com.zifferlabs.stream.SampleStream" \
/home/ubuntu/zifferlabs/src/main/java/com/zifferlabs/stream/SampleStream.java
I think it won't work since spark-submit submits your Java source code not the executable code.
Please spark-submit your application as follows:
./bin/spark-submit \
--class "com.zifferlabs.stream.SampleStream" \
/home/ubuntu/zifferlabs/target/ZifferLabs-1-jar-with-dependencies.jar
which is --class to define the "entry point" to your Spark application and the code with dependencies (as the only input argument for spark-submit).
Give it a shot and report back!