What does mean Mem Avail in yarn UI?
I set yarn.scheduler.minimum-allocation-mb to 1024 and yarn.scheduler.maximum-allocation-mb to 4096. yarn.nodemanager.resource.memory-mb is also set to -1 as default. I can see the memory is free in every nodes and UI show that Phys Mem Used is just 14%. However, the Mem Avail is 0 B and I don't know what is it and how to increase it.
I found the answer!
It's equal to yarn.nodemanager.resource.memory-mb which is The total amount of memory that YARN can use on a given node. You might need to set it higher inside yarn-site.xml depending on the amount of data you plan on processing.
The default value of this config is 8GB, although with getconf command you will see -1 which doesn't mean total memory of the system.
Before:
$ hdfs getconf -confKey yarn.nodemanager.resource.memory-mb
-1
After set it in yarn-site.xml:
$ hdfs getconf -confKey yarn.nodemanager.resource.memory-mb
40960
The result:
Related
I try hard to research Hadoop multi tenant solution and find spaceQuota is the key. After some test,I can`t really understand how hdfs calculate space quota?
I have a hdfs directory "/cb/kj_1" and set space quota of 1G,and some files of fixed size generated by dd command .
Firstly,I put four 10MB file to "/cb/kj_1",and count the space "hdfs dfs -count -q -v -h /cb/kj_1",the result as follows:
QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME
none inf 1 G 880 M 1 12 48 M /cb/kj_1
These file cost 4*10*3=120M space and 880M remaining，that`s all-right.
Then,I put two 100M file to "/cb/kj_1/",count result as follows:
QUOTA REM_QUOTA SPACE_QUOTA REM_SPACE_QUOTA DIR_COUNT FILE_COUNT CONTENT_SIZE PATHNAME
none inf 1 G 280 M 1 14 248 M /cb/kj_1
These file cost 2*100*3=600M space and 280M remaining，without a doubt.
Finally,I put one 1K size file to "/cb/kj_1",this time hdfs reject my request,and throw "org.apache.hadoop.hdfs.protocol.DSQuotaExceededException":
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.DSQuotaExceededException): The DiskSpace quota of /cb/kj_1 is exceeded: quota = 1073741824 B = 1 GB but diskspace consumed = 1182793728 B = 1.10 GB
at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyStoragespaceQuota(DirectoryWithQuotaFeature.java:214)
at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:241)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:1074)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:903)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:862)
The error message shows how many space it needs,after some calculation, (Diskspace-SPACE_QUOTA_size+REM_SPACE_QUOTA)/3=128M.
The last 1k file I put to hdfs would occupy one full block ,but the 10M file I put before cost 10M space,and 100M file cost 100M space.
It seems that one small file is not cost 128M(hdfs blocksize),if so,directory of 1G space quota can only contain 1024/3/128=2 file,but I put 6 files successfully.
The stat command shows the block size
hadoop fs -stat "%b %o" /cb/kj_1/10mb_1.dd
10485760 134217728
hadoop fs -stat "%b %o" /cb/kj_1/100mb_1.dd
104857600 134217728
I want to know how hdfs calculate the space(quota) when put files to directory ,and how it calculate space of files exists in directory.
Spark application throws this error:
> Cannot create directory
> /home/user/hdp/.sparkStaging/application_00000000_000. Name node is in
> safe mode. Resources are low on NN. Please add or free up more
> resources then turn off safe mode manually. NOTE: If you turn off
> safe mode before adding resources, the NN will immediately return to
> safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off.
So I tried this: hdfs dfs -df -h
Filesystem Size Used Available Use%
hdfs://ip-xx-xx-xxx-xxx.xxxxxxxx.xx-xxxx-x.compute.internal:xxxx 58.5 G 5.0 G 46.5 G 9%
since it has enough free space, i tried doing df -h, and found out that
${dfs.name.dir} = /xx2/namenode consumed all of free space (27G) in /xxx/xxxx3
Filesystem Size Used Avail Use% Mounted on
xxxxxxxx 3.9G 72K 3.9G 1% /xxx
xxxxx 3.9G 0 3.9G 0% /xxx/shm
/xxx/xxxx1 9.8G 3.8G 5.9G 40% /
/xxx/xxxx2 5.0G 65M 5.0G 2% /xx1
/xxx/xxxx3 27G 27G 94M 100% /xx2
I dont know how to properly free some space for that directory.
Could you check your safemode threshold value in hdfs-default.xml
dfs.safemode.threshold.pct
dfs.namenode.safemode.threshold-pct
Try increasing the threshold,then you use below to leave safemode. After that you can revert back the changes once you have found your black sheep file causing the issue
hdfs dfsadmin -safemode leave
You can check for local files(/var/log/..) as hadoop works on top of disk space of system.Also check for the directories mentioned in hadoop.tmp.dir.Use hadoop fs -rmr on all data nodes and name node to flush
I try to set the dfs.blocksize in Hadoop to 100Kb which is less than the default dfs.namenode.fs-limits.min-block-size, which is 1MB.
When I copy the file like
hdfs dfs -Ddfs.namenode.fs-limits.min-block-size=0 -Ddfs.blocksize=102400 inp.txt /input/inp.txt
I still get,
copyFromLocal: Specified block size is less than configured minimum value (dfs.namenode.fs-limits.min-block-size): 102400 < 1048576
I tried to add this property in hdfs-site.xml as well. But dfs.namenode.fs-limits.min-block-size does not seem to change.
How else would I change this property?
Try changing the value of the dfs.namenode.fs-limits.min-block-size property in the /etc/hadoop/conf/hdfs-site.xml file and restarting the NameNode, as this may be a final property which cannot be overridden by a command line setting.
I have a hadoop cluster we assuming is performing pretty "bad". The nodes are pretty beefy.. 24 cores, 60+G RAM ..etc. And we are wondering if there are some basic linux/hadoop default configuration that prevent hadoop from fully utilizing our hardware.
There is a post here that described a few possibilities that I think might be true.
I tried logging in the namenode as root, hdfs and also myself and trying to see the output of lsof and also the setting of ulimit. Here are the output, can anyone help me understand why the setting doesn't match with the open files number.
For example, when I logged in as root. The lsof looks like this:
[root#box ~]# lsof | awk '{print $3}' | sort | uniq -c | sort -nr
7256 cloudera-scm
3910 root
2173 oracle
1886 hbase
1575 hue
1180 hive
801 mapred
470 oozie
427 yarn
418 hdfs
244 oragrid
241 zookeeper
94 postfix
87 httpfs
...
But when I check out the ulimit output, it looks like this:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 806018
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I am assuming, there should be no more than 1024 files opened by one user, however, when you look at the output of lsof, there are 7000+ files opened by one user, can anyone help explain what is going on here?
Correct me if I had made any mistake understanding the relation between ulimit and lsof.
Many thanks!
You need to check limits for the process. It may be different from your shell session:
Ex:
[root#ADWEB_HAPROXY3 ~]# cat /proc/$(pidof haproxy)/limits | grep open
Max open files 65536 65536 files
[root#ADWEB_HAPROXY3 ~]# ulimit -n
4096
In my case haproxy has a directive on its config file to change maximum open files, there should be something for hadoop as well
I had a very similar issue, which caused one of the claster's YARN TimeLine server to stop due to reaching magical 1024 files limit and crashing with "too many open files" errors.
After some investigation it came out that it had some serious issues with dealing with too many files in TimeLine's LevelDB. For some reason YARN ignored yarn.timeline-service.entity-group-fs-store.retain-seconds setting (by default it's set to 7 days, 604800ms). We had LevelDB files dating back for over a month.
What seriously helped was applying a fix described in here: https://community.hortonworks.com/articles/48735/application-timeline-server-manage-the-size-of-the.html
Basically, there are a couple of options I tried:
Shrink TTL (time to live) settings First enable TTL:
<property>
<description>Enable age off of timeline store data.</description>
<name>yarn.timeline-service.ttl-enable</name>
<value>true</value>
</property>
Then set yarn.timeline-service.ttl-ms (set it to some low settings for a period of time):
\
<property>
<description>Time to live for timeline store data in milliseconds.</description>
<name>yarn.timeline-service.ttl-ms</name>
<value>604800000</value>
</property>
Second option, as described, is to stop TimeLine server, delete the whole LevelDB and restart the server. This will start the ATS database from scratch. Works fine if you failed with any other options.
To do it, find the database location from yarn.timeline-service.leveldb-timeline-store.path, back it up and remove all subfolders from it. This operation will require root access to the server where TimeLine is located.
Hope it helps.
i have a three node hadoop cluster with replication factor = 3.
Storage Directory is /app/hadoop/tmp/dfs/ for each system.
Each datanode system has hard-disk capacity of 221GB.
the Effective data of HDFS is 62GB with replication 62*3= 186GB.
Now the problem is i am falling short of storage even though i have only 186GB of data on 660 GB cluster:
HDFS shows huge difference in the space available for use:
datanode1 =7.47 GB
datanode2 =17.7 GB
datanode3 =143 GB
to make sure that these space is used by hadoop local storage, i ran this command on each datanode.
for datanode1
du -h --max-depth=1 /app/hadoop/tmp/
63G /app/hadoop/tmp/dfs
139G /app/hadoop/tmp/mapred
201G /app/hadoop/tmp/
for datanode2
du -h --max-depth=1 /app/hadoop/tmp/
126G /app/hadoop/tmp/mapred
62G /app/hadoop/tmp/dfs
188G /app/hadoop/tmp/
for datanode3
du -h --max-depth=1 /app/hadoop/tmp/dfs/
62G /app/hadoop/tmp/dfs/data
62G /app/hadoop/tmp/dfs/
here datanode1 has used 201GB space for storage.
I tried load-balancer but its showing the cluster is balanced.
here is the output.
start-balancer.sh
starting balancer, logging to /usr/lib/hadoop-0.20/logs/hadoop-ocpe-balancer-blrkec241933d.out
Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
The cluster is balanced. Exiting...
Balancing took 622.0 milliseconds
recently one of my datanode went down for few days, after fixing it this problem has arisen.
How to balance the load?
Your analysis of disk space includes MapReduce scratch directory space (/app/hadoop/tmp/mapred paths), which lie outside of HDFS and are mostly temporary data cleared upon job completion.
The DFS space (/app/hadoop/tmp/dfs) seems to be consistent with your expected usage size.
Therefore, your disk space isn't being hogged by the DataNodes, rather by the TaskTrackers - and restarting them forces a clearing of those directories.