spark-sql (hive#spark and hive#hadoop) dies with exceptions - hadoop

Question

Spark-SQL dies with the following exceptions:
Lost task 13.0 in stage 1.0 (TID 14, 10.15.0.224): java.io.InvalidClassException: org.apache.spark.sql.catalyst.expressions.AttributeMap; local class incompatible: stream classdesc serialVersionUID = -4625798594364953144, local class serialVersionUID = -4625798594364953144
and / or
com.esotericsoftware.kryo.KryoException: java.lang.ClassCastException: java.lang.Integer cannot be cast to org.apache.spark.storage.BlockManagerId
What is going on?
More details follow below.
Case: Spark-sql
single server case, local configuration
downloaded spark-1.4.0-bin-hadoop2.6 distribution
centos 6.5 Server (32 Cores, 100GB RAM)
configuration:
(variables)
export SPARK_HOME=/opt/spark-1.4.0-bin-hadoop2.6
export SPARK_MASTER_IP=127.0.0.1
export SPARK_MASTER_PORT=7077
export SPARK_MASTER_WEBUI_PORT=8080
export SPARK_LOCAL_DIRS=$SPARK_HOME/work
export SPARK_WORKER_CORES=4
export SPARK_WORKER_MEMORY=4G
export SPARK_WORKER_INSTANCES=2
export SPARK_DAEMON_MEMORY=384m
(starting and running)
$SPARK_HOME/sbin/start-all.sh
$SPARK_HOME/bin/spark-sql --master spark://127.0.0.1:7077
The data reside in *.csv.gz individual files (a few thousands of them, a few tens to hundred of megabytes each). I've tested all files for gzip/io errors (non found). The format:
A1;A2;A3;A4;A5;A6;A7;A8;A9
'string';841054310;0;11.383907333;48.788023833;380.700000000;'2014-04-28T06:11:01.753990';0;'2015-06-26T23:54:49.461211'
'string';841054310;1;11.383867000;48.788031667;381.000000000;'2014-04-28T06:14:15.272059';4.77;'2015-06-26T23:55:03.132637'
'string';841054310;2;11.383829000;48.788019000;381.000000000;'2014-04-28T06:14:18.765123';3.19;'2015-06-26T23:55:03.414938'
'string';841054310;3;11.383804667;48.788041333;380.900000000;'2014-04-28T06:14:28.477338';5.1;'2015-06-26T23:55:04.190245'
'string';841054310;4;11.383767167;48.788053167;381.000000000;'2014-04-28T06:14:31.765796';4.29;'2015-06-26T23:55:04.459112'
'string';841054310;5;11.383726500;48.788057667;381.000000000;'2014-04-28T06:14:33.778935';6.18;'2015-06-26T23:55:04.628419'
'string';841054310;6;11.383685667;48.788059333;381.000000000;'2014-04-28T06:14:35.584490';5.71;'2015-06-26T23:55:04.779281'
'string';841054310;7;11.383643667;48.788062833;381.000000000;'2014-04-28T06:14:37.289736';9.21;'2015-06-26T23:55:04.921655'
'string';841054310;8;11.383601333;48.788069333;381.100000000;'2014-04-28T06:14:38.463847';10.78;'2015-06-26T23:55:05.022049'
'string';841054310;9;11.383558000;48.788074500;381.200000000;'2014-04-28T06:14:39.570567';10.92;'2015-06-26T23:55:05.118134'
'string';841054310;10;11.383514500;48.788076000;381.200000000;'2014-04-28T06:14:40.757880';6.88;'2015-06-26T23:55:05.217862'
'string';841054310;11;11.383472000;48.788074000;381.100000000;'2014-04-28T06:14:43.364629';7.45;'2015-06-26T23:55:05.440946'
'string';841054310;12;11.383423667;48.788068333;381.100000000;'2014-04-28T06:14:44.762990';11.91;'2015-06-26T23:55:05.565626'
'string';841054310;13;11.383375833;48.788059000;381.100000000;'2014-04-28T06:14:45.762960';15.1;'2015-06-26T23:55:05.653718'
Spark-SQL commands:
create external table raw (
A1 string,
A2 int,
A3 int,
A4 double,
A5 double,
A6 double,
A7 string,
A8 double,
A9 string
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\;'
LINES TERMINATED BY '\n'
location '/directory/'
alter table raw set tblproperties('skip.header.line.count'='1');
select count(*) from raw;
Typical error:
15/07/15 20:13:22 ERROR TaskResultGetter: Exception while getting task result
com.esotericsoftware.kryo.KryoException: java.lang.ClassCastException: java.lang.Integer cannot be cast to org.apache.spark.storage.BlockManagerId
Serialization trace:
org$apache$spark$scheduler$CompressedMapStatus$$loc (org.apache.spark.scheduler.CompressedMapStatus)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:626)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
at org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:265)
at org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:95)
at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:60)
at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:51)
at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:51)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
at org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:50)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to org.apache.spark.storage.BlockManagerId
at org.apache.spark.scheduler.CompressedMapStatusFieldAccess.set(Unknown Source)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:617)
... 12 more
for a directory with a few tens of Data-files spark computes the result without an error. Once more files are added errors are thrown.
Case hive#spark:
$SPARK_HOME/sbin/start-thriftserver.sh --master spark://127.0.0.1:7077 --hiveconf hive.server2.thrift.port=10001 --driver-memory 2G
$SPARK_HOME/bin/beeline
Beeline version 1.4.0 by Apache Hive
beeline> !connect jdbc:hive2://127.0.0.1:10001
0: jdbc:hive2://127.0.0.1:10001> select count(*) from raw;
I get the following error:
Error: org.apache.spark.SparkException: Job aborted due to stage failure: Exception while getting task result: com.esotericsoftware.kryo.KryoException: Buffer underflow.
Serialization trace:
org$apache$spark$scheduler$CompressedMapStatus$$compressedSizes (org.apache.spark.scheduler.CompressedMapStatus) (state=,code=0)
Case: hive#hadoop:
Details:
single server, local mode
extracted hadoop-2.6.0 distribution under /opt
extracted apache-hive-1.2.1-bin distribution under /opt
extracted db-derby-10.11.1.1-bin distribution under /opt
configuration:
export HADOOP_HOME=/opt/hadoop-2.6.0
export DERBY_HOME=/opt/db-derby-10.11.1.1-bin
. /opt/db-derby-10.11.1.1-bin/bin/setEmbeddedCP
export HIVE_OPTS='-hiveconf mapred.job.tracker=local -hiveconf fs.default.name=file:///tmp -hiveconf hive.metastore.warehouse.dir=file:///tmp/warehouse -hiveconf javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=/tmp/metastore_db;create=true'
export HIVE_HOME=/opt/apache-hive-1.2.1-bin
export JAVA_HOME='/usr/lib/jvm/jre-1.7.0-openjdk.x86_64/'
Running the thing:
$HIVE_HOME/bin/hive
hive> select count(*) from raw;
Output (and error):
Query ID = root_20150715204331_a7a4be13-31e5-41d2-a7dd-8fdbf9d9f2b0
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Job running in-process (local Hadoop)
2015-07-15 20:44:02,482 Stage-1 map = 0%, reduce = 0%
Ended Job = job_local1201654881_0001 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
hive>
where the hack are the log information stored in this case? I cannot find anything under /tmp/hive, nor in /opt/hadoop-2.6.0/, nor in /opt/apache-hive-1.2.1-bin