We have HDP cluster version – 2.6.4
Cluster installed on redhat machines version – 7.2
We noticed about the following issue on the JournalNodes machines ( master machines )
We have 3 JournalNodes machines , and under /tmp folder we have thousands of empty folders as
drwx------. 2 hive hadoop 6 Dec 20 09:00 a962c02e-4ed8-48a0-b4bb-79c76133c3ca_resources
an also a lot of folders as
drwxr-xr-x. 4 hive hadoop 4096 Dec 12 09:02 hadoop-unjar6426565859280369566
with content as
beeline-log4j.properties BeeLine.properties META-INF org sql-keywords.properties
/tmp should be purged every 10 days according to the configuration file:
more /usr/lib/tmpfiles.d/tmp.conf
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
# See tmpfiles.d(5) for details
# Clear tmp directories separately, to make them easier to override
v /tmp 1777 root root 10d
v /var/tmp 1777 root root 30d
# Exclude namespace mountpoints created with PrivateTmp=yes
x /tmp/systemd-private-%b-*
X /tmp/systemd-private-%b-*/tmp
x /var/tmp/systemd-private-%b-*
X /var/tmp/systemd-private-%b-*/tmp
You have new mail in /var/spool/mail/root
So we decrease the retention to 1d instead of 10d in order to avoid this issue
Then indeed /tmp have only folders content of one day
But I want to ask the following questions
Is it ok to configure the retention about /tmp in Hadoop cluster to 1day ?
( I almost sure it ok , but want to hear more opinions )
Second
Why HIVE generate thousands of empty folders as XXXX_resources ,
and is it possible to solve it from HIVE service , instead to limit the retention on /tmp
It is quite normal having thousands of folders in /tmp as long as there is still free space available for normal run. Many processes are using /tmp, including Hive, Pig, etc. One day retention period of /tmp maybe too small, because normally Hive or other map-reduce tasks can run more than one day, though it depends on your tasks. HiveServer should remove temp files but when tasks fail or aborted, the files may remain, also it depend on Hive version. Better to configure some retention, because when there is no space left in /tmp, everything stops working.
Read also this Jira about HDFS scratch dir retention.
Related
I was learning hadoop and till now I configured 3 Node cluster
127.0.0.1 localhost
10.0.1.1 hadoop-namenode
10.0.1.2 hadoop-datanode-2
10.0.1.3 hadoop-datanode-3
My hadoop Namenode directory looks like below
hadoop
bin
data-> ./namenode ./datanode
etc
logs
sbin
--
--
As I learned that when we upload a large file in the cluster in divide the file into blocks, I want to upload a 1Gig file in my cluster and want to see how it is being stored in datanode.
Can anyone help me with the commands to upload file and see where these blocks are being stored.
First, you need to check if you have Hadoop tools in your path, if not - I recommend integrate them into it.
One of the possible ways of uploading a file to HDFS:hadoop fs -put /path/to/localfile /path/in/hdfs
I would suggest you read the documentation and get familiar with high-level commands first as it will save you time
Hadoop Documentation
Start with "dfs" command, as this one of the most often used commands
I am new to hadoop and recently when I was running MapReduce jobs on Openstack hadoop cluster and cd into directory on a datanode machine, I found there are two hadoop folders one is called "hadoop" while the other named"hadoop-2.7.1". Obviously, the latter one makes more sense as it tells the hadoop version. The two folder contains same sub-directories, but how these two differ from each other? What if I'd like to disable HDFS permission checking on this machine, which one should I go?
Here is a screenshot
As colors in the screenshot suggest, hadoop is not a separate directory but is just a symbolic link, obviously pointing to hadoop-2.7.1. Run ls -l to check this.
You should cd into hadoop directory. It exists intentionally to avoid writing hadoop version explicitly. When new version of hadoop is deployed, a new versioned directory will be created, and hadoop symbolic link will be changed to point to the latest versioned directory. Like this:
hadoop-2.7.1
hadoop-2.7.2
hadoop-2.7.3
hadoop -> hadoop-2.7.3
We are using HDP hadoop distribution v2.3.2, we are dealing with Hive external tables and these are queried on daily basis.
After few days since process started, the data directories contain lot of staging directories with format: .hive-staging_hive_date-time_
There are lot of staging directories generated, each staging directory corresponds to a query run on the Hive table.
What can I do to avoid these staging directories to be piled up into my data directories ?
The answer I posted at https://stackoverflow.com/a/35583367/14186 may help you here. You can configure Hive to make those staging dirs some place else (normally they are made as a subdir of the final destination dir)
In the example from that answer, I have hive put them in dirs under /tmp, and we have a cron-job that we run each day to delete any leftover staging dirs older than 1 week to keep things tidy in case hive doesn't remove them.
I've searched and I've been reading on Cloudera Hadoop on removing mount point file systems but I cannot find a thing on removing them.
I have two SSD drives in 6 machines and when I initially installed Cloudera Hadoop it added all file systems and I only need two mount points to run a few teragen and terasorts.
I need to remove everything except for:
/dev/nvme0n1 and /dev/nvme1n1
In Cloudera Manager you can modify the list of drives used for HDFS data at:
Clusters > HDFS > Configuration > DataNode Default Group (or whatever you may have renamed this to) > DataNode Data Directory
I was going through the HADOOP fs commands list. I am little perplexed not to find any "cd" command in hadoop fs.
Why is it so? It might sound silly question for the HADOOP users, but as I am beginner I can not understand why there is no list of cd command in HADOOP fs level?
Think about it like this:
Hadoop has a special file system called "hdfs" which runs on top of existing say linux file system. There is no concept of current or present working directory a.k.a. pwd
Let's say we have following structure in hdfs:
d1/
d2/
f1
d3/
f2
d4/
f3
You could do cd in your Linux file system from moving from one to the other but do you think changing directory in hadoop would makes sense? HDFS is like virtual file system and you dont directly interact with hdfs except via hadoop command or job tracker.
HDFS provides various features that enable accessing HDFS(Hadoop Filesystem) easy on local machines or edge nodes. You have an option to mount HDFS using any of the following methods. Once Hadoop file system is mounted on your machine, you may use cd command to browse through the file system (It's is like mounting remote network filesystem like NAS)
Fuse dfs (Available from Hadoop 0.20 onwards )
NFSv3 Gateway access to HDFS data (Available from Hadoop version
Hadoop 2.2.0)