Spark 1.3 and Cassandra 3.0 problems with guava - maven

Question

I am trying to connect to Cassandra 3.0 from Spark 1.3. I know that there is Cassandra connector for each version in spark, but spark-cassandra-connector-java_2.10:1.3.0 connector depends on cassandra-driver-core:2.1.5, that's why I am using the latest cassandra connector which depends the latest core driver. Anyway, so far this was not the problem. The problem is the com.google.guava package I suppose.
My pom looks like this:
...
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector-java_2.10</artifactId>
<version>1.5.0-M3</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>1.5.0-M3</version>
</dependency>
...
I have excluded google guava from everywhere with:
<exclusions>
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
</exclusions>
so in the dependency tree only this is present com.google.guava:guava:jar:16.0.1 under com.datastax.spark:spark-cassandra-connector-java_2.10:jar:1.5.0-M3:compile.
However I am still getting the following error:
yarn.ApplicationMaster: User class threw exception: Failed to open native connection to Cassandra at {139.19.52.111}:9042
java.io.IOException: Failed to open native connection to Cassandra at {139.19.52.111}:9042
at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:162)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31)
at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:56)
at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81)
at com.ambiverse.tagging.dao.impl.DAOCassandra.createTable(DAOCassandra.java:45)
at com.ambiverse.tagging.dao.impl.DAOCassandra.createTable(DAOCassandra.java:64)
at com.ambiverse.tagging.dao.impl.DAOCassandra.savePairRDD(DAOCassandra.java:70)
at com.ambiverse.tagging.statistics.entitycorrelation.CorrelationStatisticsSparkRunner.run(CorrelationStatisticsSparkRunner.java:176)
at com.ambiverse.tagging.statistics.entitycorrelation.CorrelationStatisticsSparkRunner.main(CorrelationStatisticsSparkRunner.java:94)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)
Caused by: java.lang.NoSuchMethodError: com.google.common.util.concurrent.Futures.withFallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureFallback;Ljava/util/concurrent/Executor;)Lcom/google/common/util/concurrent/ListenableFuture;
at com.datastax.driver.core.Connection.initAsync(Connection.java:178)
at com.datastax.driver.core.Connection$Factory.open(Connection.java:742)
at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:240)
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:187)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:79)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1393)
at com.datastax.driver.core.Cluster.getMetadata(Cluster.java:402)
at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155)
Before somebody point me to this blog post for solution: http://arjon.es/2015/10/12/making-hadoop-2-dot-6-plus-spark-cassandra-driver-play-nice-together/, I am using maven as a build tool, not sbt. If you know how can I do the exact same thing with maven, that would be great.

Answer 1

Although i work with scala + sbt, i had several mismatch between different artifacts with spark, and one among them is guava.
here is how i solved it (dependencies in sbt):
val sparkVersion = "1.6.1"//"2.0.0-preview"//
val sparkCassandraConnectorVersion = "1.6.0"
val scalaGuiceVersion = "4.0.1"
val cassandraUnitVersion = "3.0.0.1"
val typesafeConfigVersion = "1.3.0"
val findbugsVersion = "3.0.0"
val sparkRabbitmqVersion = "0.4.0.20160613"
val nettyAllVersion = "4.0.33.Final"
val guavaVersion = "19.0"
val jacksonVersion = "2.7.4"
val xbeanAsm5ShadedVersion = "4.5"
val commonsBeanutilsVersion = "1.8.0"
//IMPORTANT: all spark dependency magic is done in one place, to overcome the assembly mismatch errors
val sparkDependencies :List[ModuleID] = List(
("org.apache.spark" %% "spark-core" % sparkVersion).exclude("com.esotericsoftware.minlog", "minlog"),
"org.apache.spark" %% "spark-sql" % sparkVersion,
"org.apache.spark" %% "spark-streaming" % sparkVersion,
("com.datastax.spark" %% "spark-cassandra-connector"
% sparkCassandraConnectorVersion).exclude("org.apache.cassandra", "cassandra-clientutil"),
"com.stratio.receiver" % "spark-rabbitmq_1.6" % sparkRabbitmqVersion,//"0.3.0-b", //,//
"org.scalatest" %% "scalatest" % scalaTestVersion % "test",
"org.apache.xbean" % "xbean-asm5-shaded" % xbeanAsm5ShadedVersion,//,//, //https://github.com/apache/spark/pull/9512/files
"io.netty" % "netty-all" % nettyAllVersion,
"commons-beanutils" % "commons-beanutils" % commonsBeanutilsVersion,
"com.google.guava" % "guava" % guavaVersion,
"com.fasterxml.jackson.module" %% "jackson-module-scala" % jacksonVersion,//fix jackson mismatch problem
"com.fasterxml.jackson.core" % "jackson-databind" % jacksonVersion,//fix jackson mismatch problem
//override findbugs artifacts versions(fix assembly issues)
"com.google.code.findbugs" % "annotations" % findbugsVersion,
"com.google.code.findbugs" % "jsr305" % findbugsVersion
).map(_.exclude("commons-collections", "commons-collections"))
i hope it will help.