In the 50070 monitor web page,It shows:
16,889,283 files and directories, 9,236,314 blocks = 26,125,597 total filesystem object(s).
Heap Memory used 14.84 GB of 25.85 GB Heap Memory. Max Heap Memory is 25.85 GB.
confusion-over-hadoop-namenode-memory-usage tells me how to calculate the memory of namenode used.I find the method is not for me. 26,125,597 * 150byte ≈ 4G,but i use near 15G memory!
But,the standby namenode just use 5G memory.
16,889,283 files and directories, 9,236,314 blocks = 26,125,597 total filesystem object(s).
Heap Memory used 4.9 GB of 25.85 GB Heap Memory. Max Heap Memory is 25.85 GB.
Related
I have a program which accesses a large memory block allocated with mmap. It accesses it unevenly, mostly accessing the first ~1 GB on memory, sometimes the next ~2 GB of memory, and rarely the last ~4 GB of memory. The memory is a shared memory mapping with PROT_READ and PROT_WRITE backed by an unlinked file.
Compared to the Linux version, I've found the MacOS version is exceedingly slow. Yet, the memory pressure is low. (6.42 Used, 9.51 Cached.)
The following usage statistics originate from activity monitor:
"Memory": 1.17 GB
Real memory Size: 3.71 GB
Virtual Memory Size: 51.15 GB
Shared Memory Size: 440 KB
Private Memory Size: 3.74 GB
Why is this? Is there anyway to improve caching behavior?
How can I determine the heap size required for 1 GB logs having 1 day retention period?
if I take the machine with 32 GB heap size (64 GB RAM) how many GB logs I can keep in this for 1 day?
It depends on various factors like the number of indexing requests, search requests, cache utilization, size of search and indexing requests, number of shards/segments etc, also heap size should follow the sawtooth pattern, and instead of guessing it, you should start measuring it.
The good thing is that you can starting right, by assigning 50% of RAM as ES Heap size which is not crossing 32 GB.
I am trying to size our pods using the actuator metrics info. With the below K8 resource quota configuration;
resources:
requests:
memory: "512Mi"
limits:
memory: "512Mi"
We are observing that jvm.memory.max returns ~1455 mb. I understand that this value includes heap and non-heap. Further drilling into the api (jvm.memory.max?tag=area:nonheap) and (jvm.memory.max?tag=area:heap) results in ~1325mb and ~129mb respectively.
Obviously with the non-heap set to max out at a value greater than the K8 limit, the container is bound to get killed eventually. But why is the jvm (non-heap memory) not bounded by the memory configuration of the container (configured in K8)?
The above observations are valid with java 8 and java 11. The below blog discusses the experimental options with java 8 where CPU and heap configurations are discussed but no mention of non-heap. What are some suggestions to consider in sizing the pods?
-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
Source
Java 8 has a few flags that can help the runtime operate in a more container aware manner:
java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -jar app.jar
Why you get maximum JVM heap memory of 129 MB if you set the maximum container memory limit to 512 MB? So the answer is that memory consumption in JVM includes both heap and non-heap memory. The memory required for class metadata, JIT complied code, thread stacks, GC, and other processes is taken from the non-heap memory. Therefore, based on the cgroup resource restrictions, the JVM reserves a portion of the memory for non-heap use to ensure system stability.
The exact amount of non-heap memory can vary widely, but a safe bet if you’re doing resource planning is that the heap is about 80% of the JVM’s total memory. So if you set the set maximum heap to 1000 MB, you can expect that the whole JVM might need around 1250 MB.
The JVM read that the container is limited to 512M and created a JVM with maximum heap size of ~129MB. Exactly 1/4 of the container memory as defined in the JDK ergonomic page.
If you dig into the JVM Tuning guide you will see the following.
Unless the initial and maximum heap sizes are specified on the command line, they're calculated based on the amount of memory on the machine. The default maximum heap size is one-fourth of the physical memory while the initial heap size is 1/64th of physical memory. The maximum amount of space allocated to the young generation is one third of the total heap size.
You can find more information about it here.
In Cassandra configuration memory limit is given.Suppose we have physical memory of 8 gb out of which 2 gb is allocated to Cassandra. So if Cassandra's memory usage goes upto 2gb, does additional memory from 8 gb will be allocated to Cassandra or not?
We use ES to store around 2.5TB of data. We have 12 primary shards and 2 replica shards.
We are currently load testing ES and I read the following article
https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html
This article states 2 important things. First allocate 50% of Memory to Lucene and Second Don't cross 30.5GB limit for heap space.
I don't clearly understand the 30.5GB limit. I understand that if I am to set 40GB over 30.5 GB i will loose more than i gain(because of compressed pointers) but say if i have hardware of around 250GB RAM what are the reasons that i should only allocate 30.5GB and not 120GB for heap. Won't i start seeing gains after 70-80GB heap setting over 30.5 GB heap. Can somebody list down all the reasons?