Any tips for making Hive run faster on Hadoop? - performance

Question

I'm new to Hive and Hadoop. I have Hadoop configured for pseudo-distributed operation with one data node and one name node, all on localhost.
I have a trivial employee table containing 4 records. I can select the records in a reasonable amount of time, but anything beyond that takes really long. For example:
0: jdbc:hive2://localhost:10000> select * from emp;
+------------+------------+-------------+-------------+------------+
| emp.empno | emp.ename | emp.job | emp.deptno | emp.etype |
+------------+------------+-------------+-------------+------------+
| 7369 | SMITH | CLERK | 10 | PART_TIME |
| 7400 | JONES | ENGINEER | 10 | FULL_TIME |
| 7500 | BROWN | NIGHTGUARD | 20 | FULL_TIME |
| 7510 | LEE | ENGINEER | 20 | FULL_TIME |
+------------+------------+-------------+-------------+------------+
4 rows selected (0.643 seconds)
0: jdbc:hive2://localhost:10000> select * from emp order by empno;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
+------------+------------+-------------+-------------+------------+
| emp.empno | emp.ename | emp.job | emp.deptno | emp.etype |
+------------+------------+-------------+-------------+------------+
| 7369 | SMITH | CLERK | 10 | PART_TIME |
| 7400 | JONES | ENGINEER | 10 | FULL_TIME |
| 7500 | BROWN | NIGHTGUARD | 20 | FULL_TIME |
| 7510 | LEE | ENGINEER | 20 | FULL_TIME |
+------------+------------+-------------+-------------+------------+
4 rows selected (225.852 seconds)
What's it doing that's taking so long? Are there polling periods that I could reduce? I know that Hive isn't optimized for small tasks but this seems ridiculous.
Here's the various files:
hive-site.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hive.mapred.mode</name>
<value>nostrict</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:derby:;databaseName=metastore_db;create=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.apache.derby.jdbc.EmbeddedDriver</value>
</property>
<property>
<name>hive.exec.local.scratchdir</name>
<value>/home/hadoop/tmp</value>
</property>
<property>
<name>hive.downloaded.resources.dir</name>
<value>/home/hadoop/tmp/${hive.session.id}_resources</value>
</property>
<property>
<name>hive.querylog.location</name>
<value>/home/hadoop/tmp</value>
</property>
<property>
<name>hive.server2.logging.operation.log.location</name>
<value>/home/hadoop/tmp/operation_logs</value>
</property>
</configuration>
hdfs.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>file:///home/hadoop/hadoopinfra/hdfs/namenode</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>file:///home/hadoop/hadoopinfra/hdfs/datanode</value>
</property>
</configuration>
core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.proxyuser.hive.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hive.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
</configuration>
mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
yarn-site.xml
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

Answer 1

Use beeline shell, hive is deprecated.

Answer 2

You are using 'order by' in query, order by imposes the total order of all results, so there has to be single reducer to sort the final output.
Though the number of records are less in this case, because of the two stages (Map and Reduce) and Disk I/O, the query took time to process. If the number of rows in the output is too large, the single reducer could take a very long time to finish.
Suggest you to run Hive on Tez or Spark engines and dont run the queries which require full scan tables unless is really needed.

Answer 3

Your hive is using Map Reduce for processing the data which is deprecated in Hive2 and also giving slow response so you have to set Tez or Spark as your hive execution engine which gives you faster result.
As per my experience I suggest You have to used Tez as execution engine for best performance. For setup Hive-Tez please flow this document.
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_command-line-installation/content/ref-d677ca50-0a14-4d9e-9882-b764e689f6db.1.html

Answer 4

You can do two things to run your query faster :
1. Use the combination of DISTRIBUTE BY and SORT BY. DISTRIBUTE BY will ensure that all keys with a certain value will end up on the same data node. SORT BY will then sort the data on each node.
2. set the execution engine to Tez before executing your query.
set hive.execution.engine=tez;
I think this will certainly make your query run faster.

Answer 5

Set hive.fetch.task.conversion=none;
Set this property and run it again

Answer 6

Use following at the start of the hive query:
-- use cost-based query optimization
set hive.cbo.enable=true;
set hive.compute.query.using.stats=true;
set hive.stats.fetch.column.stats=true;
set hive.stats.fetch.partition.stats=true;
-- use dynamic hive partition
set hive.exec.dynamic.partition = true;
set hive.exec.dynamic.partition.mode = nonstrict;
Please add comments, if you like my answer!