Issue on configure hive on spark - hadoop

Question

I have downloaded spark-2.0.0-bin-hadoop2.7. Could any one advise how to configure hive on this and use in scala console? Now I am able to run RDD's on file using Scala (spark-shell console).

Answer 1

Follow the official Hive on Spark documentation:
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
You can set on Hive the spark engine by using the following command:
set hive.execution.engine=spark;
or by adding it on hive-site.xml (refer to kanishka post)
Then prior to Hive 2.2.0, copy the spark-assembly jar to HIVE_HOME/lib.
Since Hive 2.2.0, Hive on Spark runs with Spark 2.0.0 and above, which doesn't have an assembly jar.
To run with YARN mode (either yarn-client or yarn-cluster), copy the following jars to HIVE_HOME/lib.
scala-library
spark-core
spark-network-common
Set the spark_home:
export $SPARK_HOME=/path-to-spark
Start Spark Master and Workers:
spark-class org.apache.spark.deploy.master.Master
spark-class org.apache.spark.deploy.worker.Worker spark://MASTER_IP:PORT
Configure Spark:
set spark.master=<Spark Master URL>;
set spark.executor.memory=512m;
set spark.yarn.executor.memoryOverhead=10~20% of spark.executor.memory(value);
set spark.serializer=org.apache.spark.serializer.KryoSerializer;

Answer 2

Put your hive-site.xml on spark conf directory

Answer 3

Hive can support multiple execution engine. Like TEZ, Spark.
You can set the property in hive-site.xml
</property>
<name>hive.execution.engine</name>
<value>spark</value>
<description>
I am choosing Spark as the execution engine
</description>
</property>
Copy jars spark-assembly jar to HIVE_HOME/lib
Set the spark_home
set the below properties
set spark.master=<Spark Master URL>
set spark.eventLog.enabled=true;
set spark.eventLog.dir=<Spark event log folder (must exist)>
set spark.executor.memory=512m;
set spark.serializer=org.apache.spark.serializer.KryoSerializer;
Above steps would suffice i think