How spark2.0 to connect to hive-jdbc-1.1.0 - jdbc

Question

I have a question.When i run my spark task , with hive-jdbc in local ,i can connect the hive .But when i run it in cluster with spark-submit,it failed .
Exception in thread "main" java.sql.SQLException: Could not establish connection to jdbc:hive2://172.16.28.99:10000/vdm_da_dev.db: Required field 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null, configuration:{use:database=vdm_da_dev.db})"
The client is hive-jdbc-1.1.0-cdh5.6.0,and the server is hive-1.1.0-cdh5.6.0.
But the spark is spark-2.0.0,the hive-jdbc in /opt/spark/jars is hive-jdbc-1.2.1.spark2.jar.
I replace it with hive-jdbc-1.1.0-cdh5.6.0.jar of all nodes,but still the wrong .
I package the project with dependencies,but in the spark-submit,it did not use them.
How can i do it?
Please,anyone can help me . Thanks very much.

Answer 1

You can use guide from cloudera for additional parameters:
running spark apps on cloudera
In general spark-submit should look like:
spark-submit --class *class_main* \
--master yarn \
--deploy-mode cluster \
--conf "key=value" \
--files path_to_spark_conf/hive-site.xml \
--jars full_path/addiational_dependecy_jars \
app_package.jar
In --jars possibly you should provide datanucleus-core, datanucleus-rdbms, datanucleus-api-jdo in order to work with hive and hive metastore.

Answer 2

Thanks FaigB very much!
I did as #FaigB told,but i found there were no the dependence jars in dependency classpath.
Then i removed all the hive dependence jars in /opt/spark/jars/,and copy them from /opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/jars/.Then i solved the mistake of mismatched version.
Just like:
-rw-r--r--. 1 spark spark 138464 8月 19 13:55 hive-beeline-1.2.1.spark2.jar_bk
-rw-r--r--. 1 spark spark 37995 11月 15 01:00 hive-cli-1.1.0-cdh5.6.0.jar
-rw-r--r--. 1 spark spark 40817 8月 19 13:55 hive-cli-1.2.1.spark2.jar_bk
-rw-r--r--. 1 spark spark 11498852 8月 19 13:55 hive-exec-1.2.1.spark2.jar_bk
-rw-r--r--. 1 spark spark 95006 11月 9 14:59 hive-jdbc-1.1.0-cdh5.6.0.jar
-rw-r--r--. 1 spark spark 100680 8月 19 13:55 hive-jdbc-1.2.1.spark2.jar_bk
-rw-r--r--. 1 spark spark 5505200 8月 19 13:55 hive-metastore-1.2.1.spark2.jar_bk
-rw-r--r--. 1 spark spark 1874242 11月 15 01:00 hive-service-1.1.0-cdh5.6.0.jar
-rw-r--r--. 1 spark spark 1033763 8月 19 13:55 spark-hive_2.11-2.0.0.jar_bk
-rw-r--r--. 1 spark spark 1813851 8月 19 13:55 spark-hive-thriftserver_2.11-2.0.0.jar_bk
[spark#d2 jars]$ cp hive-service-1.1.0-cdh5.6.0.jar /opt/spark/jars/
[spark#d2 jars]$ cp hive-cli-1.1.0-cdh5.6.0.jar /opt/spark/jars/
[spark#d2 jars]$ cp spark-hive_2.10-1.5.0-cdh5.6.0.jar /opt/spark/jars/
[spark#d2 jars]$ cp hive-exec-1.1.0-cdh5.6.0.jar /opt/spark/jars/
[spark#d2 jars]$ cp hive-metastore-1.1.0-cdh5.6.0.jar /opt/spark/jars/
[spark#d2 jars]$ cp hive-beeline-1.1.0-cdh5.6.0.jar /opt/spark/jars/