Executing a long running Hive Tez query, it rarely fails with:
java.io.IOException: File hdfs://XXX with newer attempt ID 1 is smaller than the file hdfs://YYY with older attempt ID 0
In our 20 node HDP 3.1.5 cluster (Hive 3.1.0 and Tez 0.9.1), it fails once over around 200 executions
We where hitting HIVE-23354
It seems to have no workaround. It is solved at Hive 4.0.0
I had the same issue with a query with lots of big joins. Decresing (512 mb->16 mb in my case) the size of the tables that fit in memory, namely hive.auto.convert.join.noconditionaltask.size solved the problem for me.
Stack: HDP 3.1.4, Tez 0.9.1, Hive 3.1.0.
Related
I am using Hadoop - 1.0.4 & Hive - 1.2.1.
I am facing issue with select query in hive CLI. snippet of error log attached. Please help me resolving the issue.
Thanks Nirmal. Its resolved after upgrading hadoop version to 2.6.0
I have upgraded HortonWorks from 2.2 to 2.3 for our Hadoop cluster and made all the required changes to Hue (given in HortonWorks documentation) but hue client has following issues when accessing Tez on Hive browser whereas Tez on Hive CLI works perfectly alright. Earlier (HDP 2.2), Tez was compatible with Hue but is there a problem with Hue client with Tez on HDP 2.3?
Issue 1: After upgrading HDP 2.2 to 2.3, Tez still looks for HDP 2.2 libs files in HDFS and local location.
HDP 2.2 location:
HDFS: /hdp/apps/2.2.9.0-3393
Local Files: /usr/hdp/2.2.9.0-3393
Temporary Solution to Issue 1: Moved 2.3 supporting files to 2.2
HDFS:
hdfs dfs -cp /hdp/apps/2.3.2.0-2950/tez/tez.tar.gz /hdp/apps/2.2.9.0-3393/tez/
Local Files:
cp /usr/hdp/2.3.2.0-2950/hive/lib/hive-exec-0.14.0.2.2.9.0-3393.jar /usr/hdp/2.2.9.0-3393/hive/lib/
cp /usr/hdp/2.3.2.0-2950/hadoop/lib/jersey*.jar /usr/hdp/2.2.9.0-3393/hadoop/lib/
cp /usr/hdp/2.3.2.0-2950/hadoop-yarn/lib/jersey*.jar /usr/hdp/2.2.9.0-3393/hadoop-yarn/lib/
cp /usr/hdp/2.3.2.0-2950/hadoop-mapreduce/lib/jersey*.jar /usr/hdp/2.2.9.0-3393/hadoop-mapreduce/lib/
Technically, Tez must look for "/usr/hdp/current" directory which is 2.3.2.0-2950.
Issue 2: Running Hive on Tez through Hue gives following error:
Error:
Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
ERROR : Failed to execute tez graph.
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No cluster diagnostics found.
Through some research it is found that when Tez execution is not required to run a Hive query, Hue client version matches with AM version whereas any query which needs tez execution Hue client version shown as unknown.
Client version and AM version match when Tez execution is not required:
Created DAGAppMaster for application appattempt_1470224940790_0082_000001, versionInfo=[ component=tez-dag, version=0.7.0.2.3.2.0-2950, revision=4900a9cea70487666ace4c9e490d4d8fc1fee96f, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150930-1859 ] [INFO] [main] |app.DAGAppMaster|: Comparing client version with AM version, clientVersion=0.7.0.2.3.2.0-2950, AMVersion=0.7.0.2.3.2.0-2950
Client version and AM version doesn't match when Tez execution is enabled:
Created DAGAppMaster for application appattempt_1470224940790_0092_000001, versionInfo=[ component=tez-dag, version=0.7.0.2.3.2.0-2950, revision=4900a9cea70487666ace4c9e490d4d8fc1fee96f, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150930-1859 ]
Comparing client version with AM version, clientVersion=Unknown, AMVersion=0.7.0.2.3.2.0-2950
[ERROR] [main] |app.DAGAppMaster|: Incompatible versions found, clientVersion=Unknown, AMVersion=0.7.0.2.3.2.0-2950
Can anyone help on how to find a solution for incompatible version error when Tez is enabled through Hue on HDP 2.3.
I have setup presto 0.76, Cassandra 2.1.2 and created a mykeyspace and a table to it. I started both the Cassandra daemons and Presto daemons. When I try to query Cassandra using presto CLI it returns
presto:mykeyspace> select * from userinfo;
Query 20141216_181006_00021_me4u4 failed: replicate_on_write is not a column defined in this metadata
So is there any way to get over it?
Use latest version 0.88 with fixes for cassandra, http://prestodb.io/docs/current/release/release-0.88.html
I have configured hadoop 2.2.0 as single node cluster ( was able to run example jar)
Now I need to make hive perform queries using this hadoop
should I set
mapred.job.tracker
to
yarn.resourcemanager.resource-tracker.address
property?
tried so, but can't see the data loaded into hive tables in hdfs
I don't have enough reputation points to add a comment, so trying to help via an answer.
What are the daemons currently running for Hadoop? Use ps -eaf |
grep "java" to check.
Do you see the JobTracker running or the ResourceManager?
Also, can you elaborate on the steps you performed to install Hive?
I have screen cast, Installing Apache Hive that walks you through installing Hive. Next, you can follow my blog post Apache Hive - Getting Started. Hope this helps.
I am executing a simple Hadoop MapReduce program with HBase as an input and output.
I am getting the error:
java.lang.RuntimeException: org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for OutPut,,99999999999999 after 10 tries.
This exception appeared to us when there was difference in hbase version.
Our code was built with and running with 0.94.X version of hbase jars. Whereas the hbase server was running on 0.90.3.
When we changed our pom file with right version (0.90.3) of hbase jars it started working fine.
Query bin/hbase hbck and find in which machine Region server is running.
Make sure that all your Region server is up and running.
Use start regionserver for starting Region server
Even if Regionserver at the machine is started it may fail because of time sync.
Make sure you have NTP installed on all Regionserver nodes and HbaseMaster node.
As Hbase works on a key-value pair where it uses the Timestamp as the Index, So it allows a time skew less than 3 seconds.
Deleting (or move to /tmp) the WAL logs helped in our case:
hdfs dfs -mv /apps/hbase/data/MasterProcWALs/state-*.log /tmp