We use ES to store around 2.5TB of data. We have 12 primary shards and 2 replica shards.
We are currently load testing ES and I read the following article
https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html
This article states 2 important things. First allocate 50% of Memory to Lucene and Second Don't cross 30.5GB limit for heap space.
I don't clearly understand the 30.5GB limit. I understand that if I am to set 40GB over 30.5 GB i will loose more than i gain(because of compressed pointers) but say if i have hardware of around 250GB RAM what are the reasons that i should only allocate 30.5GB and not 120GB for heap. Won't i start seeing gains after 70-80GB heap setting over 30.5 GB heap. Can somebody list down all the reasons?
Related
How can I determine the heap size required for 1 GB logs having 1 day retention period?
if I take the machine with 32 GB heap size (64 GB RAM) how many GB logs I can keep in this for 1 day?
It depends on various factors like the number of indexing requests, search requests, cache utilization, size of search and indexing requests, number of shards/segments etc, also heap size should follow the sawtooth pattern, and instead of guessing it, you should start measuring it.
The good thing is that you can starting right, by assigning 50% of RAM as ES Heap size which is not crossing 32 GB.
I have implemented chronicle map as:
ChronicleMapBuilder
.of(LongValue.class, ExceptionQueueVOInterface.class)
.name(MAP_NAME)
.entries(2_000_000)
.maxBloatFactor(3)
.create();
Map has around 3.5 million records. I am using this map to server as cache for search operation. Many times to search for list of entries we have to iterate over whole map.
3.5 Million records taking 8.7 GB of RAM and my java heap allocated memory is 36 GB. Total RAM in server is 49 GB.
When i am iterating map using below code:
map.forEachEntry(entry -> {
if(exportSize.get() != MAX_EXPORT_SIZE && TeamQueueFilter.applyFilter(exceptionFilters, entry.value().get(), queType, uid)){
ExceptionQueue exceptionQueue = getExceptionQueue(entry.value().get());
if(exceptionFilters.isRts23Selected() || exceptionQueue.getMessageType().equals(MessageType.RTS23.toString())){
exceptionQueue.setForsId(Constant.BLANK_STRING);
rts23CaseIdMap.put(String.valueOf(entry.key().get().getValue()), exceptionQueue);
}else {
handleMessage(exceptionQueue, entry.key().get(), caseDetailsList);
}
exportSize.incrementAndGet();
}
});
it always gives me memory error. VM.array size is less than available memory.
Any tips here to iterate over this large map. Thanks for help.
I have to guess as you haven't provided any error here that the error message you are getting says something like java.lang.OutOfMemoryError: Direct buffer memory.
Chronicle Map will not allocate its contents on heap, instead it uses offheap memory-mapping, and likely you're running out of offheap space. You need to check what is the size of the Chronicle Map's file on the disk, the same amount of the offheap memory will be used for mmapping.
Offheap memory can be adjusted using the -XX:MaxDirectMemorySize flag.
I am trying to size our pods using the actuator metrics info. With the below K8 resource quota configuration;
resources:
requests:
memory: "512Mi"
limits:
memory: "512Mi"
We are observing that jvm.memory.max returns ~1455 mb. I understand that this value includes heap and non-heap. Further drilling into the api (jvm.memory.max?tag=area:nonheap) and (jvm.memory.max?tag=area:heap) results in ~1325mb and ~129mb respectively.
Obviously with the non-heap set to max out at a value greater than the K8 limit, the container is bound to get killed eventually. But why is the jvm (non-heap memory) not bounded by the memory configuration of the container (configured in K8)?
The above observations are valid with java 8 and java 11. The below blog discusses the experimental options with java 8 where CPU and heap configurations are discussed but no mention of non-heap. What are some suggestions to consider in sizing the pods?
-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
Source
Java 8 has a few flags that can help the runtime operate in a more container aware manner:
java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -jar app.jar
Why you get maximum JVM heap memory of 129 MB if you set the maximum container memory limit to 512 MB? So the answer is that memory consumption in JVM includes both heap and non-heap memory. The memory required for class metadata, JIT complied code, thread stacks, GC, and other processes is taken from the non-heap memory. Therefore, based on the cgroup resource restrictions, the JVM reserves a portion of the memory for non-heap use to ensure system stability.
The exact amount of non-heap memory can vary widely, but a safe bet if you’re doing resource planning is that the heap is about 80% of the JVM’s total memory. So if you set the set maximum heap to 1000 MB, you can expect that the whole JVM might need around 1250 MB.
The JVM read that the container is limited to 512M and created a JVM with maximum heap size of ~129MB. Exactly 1/4 of the container memory as defined in the JDK ergonomic page.
If you dig into the JVM Tuning guide you will see the following.
Unless the initial and maximum heap sizes are specified on the command line, they're calculated based on the amount of memory on the machine. The default maximum heap size is one-fourth of the physical memory while the initial heap size is 1/64th of physical memory. The maximum amount of space allocated to the young generation is one third of the total heap size.
You can find more information about it here.
elasticsearch 1.7.2 on CentOS, 8GB RAM, multi node cluster.
We are reviewing: https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html
and have a few questions.
1) Other than the breakers, how do we see approaching max-out of fielddata RAM ? (The stats in /_stats/fielddata?fields=* , like memory_size, don't show 'available_memory')
2) When we run /_stats/fielddata?fields=*, we see "primaries" and "total", which is fine. But what is the total a total of? It appears to be memory_size * number of nodes, (but that doesn't make sense to us).
3) We moved from nodes with 4GB RAM to nodes with 8GB RAM, yet we feel like the avail RAM (on stock elasticsearch.yml) for fielddata did not double. Is that a fair conclusion?
The essential thing we want to monitor is: How close are we to maxing RAM... but we don't see what to measure.
First of all, you are not maxing RAM, but HEAP. ES is using both Heap memory (from the JVM) and, also, RAM (as file data cache).
To see how much heap your nodes are using, this is the command to use: GET /_nodes/stats/jvm, which will output something like
"jvm": {
"timestamp": 1447790529519,
"uptime_in_millis": 84853,
"mem": {
"heap_used_in_bytes": 311374768,
"heap_used_percent": 14,
"heap_committed_in_bytes": 2112618496,
You need 100 * heap_used_in_bytes / heap_committed_in_bytes and resulting a percentage of usage. Or simply ysing heap_used_percent. For the actual RAM usage, use your OS reports for that or GET /_nodes/stats/os.
For fielddata you can get statistics per node using GET /_nodes/stats/indices/fielddata
I am trying to understand how heap fragmenation works. What does the following output tell me?
Is this heap overly fragmented?
I have 243010 "free objects" with a total of 53304764 bytes. Are those "free object" spaces in the heap that once contained object but that are now garabage collected?
How can I force a fragmented heap to clean up?
!dumpheap -type Free -stat
total 243233 objects
Statistics:
MT Count TotalSize Class Name
0017d8b0 243010 53304764 Free
It depends on how your heap is organized. You should have a look at how much memory in Gen 0,1,2 is allocated and how much free memory you have there compared to the total used memory.
If you have 500 MB managed heap used but and 50 MB is free then you are doing pretty well. If you do memory intensive operations like creating many WPF controls and releasing them you need a lot more memory for a short time but .NET does not give the memory back to the OS once you allocated it. The GC tries to recognize allocation patterns and tends to keep your memory footprint high although your current heap size is way too big until your machine is running low on physical memory.
I found it much easier to use psscor2 for .NET 3.5 which has some cool commands like ListNearObj where you can find out which objects are around your memory holes (pinned objects?). With the commands from psscor2 you have much better chances to find out what is really going on in your heaps. Most commands are also available in SOS.dll in .NET 4 as well.
To answer your original question: Yes free objects are gaps on the managed heap which can simply be the free memory block after your last allocated object on a GC segement. Or if you do !DumpHeap with the start address of a GC segment you see the objects allocated in that managed heap segment along with your free objects which are GC collected objects.
This memory holes do normally happen in Gen2. The object addresses before and after the free object do tell you what potentially pinned objects are around your hole. From this you should be able to determine your allocation history and optimize it if you need to.
You can find the addresses of the GC Heaps with
0:021> !EEHeap -gc
Number of GC Heaps: 1
generation 0 starts at 0x101da9cc
generation 1 starts at 0x10061000
generation 2 starts at 0x02aa1000
ephemeral segment allocation context: none
segment begin allocated size
02aa0000 02aa1000** 03836a30 0xd95a30(14244400)
10060000 10061000** 103b8ff4 0x357ff4(3506164)
Large object heap starts at 0x03aa1000
segment begin allocated size
03aa0000 03aa1000 03b096f8 0x686f8(427768)
Total Size: Size: 0x115611c (18178332) bytes.
------------------------------
GC Heap Size: Size: 0x115611c (18178332) bytes.
There you see that you have heaps at 02aa1000 and 10061000.
With !DumpHeap 02aa1000 03836a30 you can dump the GC Heap segment.
!DumpHeap 02aa1000 03836a30
Address MT Size
...
037b7b88 5b408350 56
037b7bc0 60876d60 32
037b7be0 5b40838c 20
037b7bf4 5b408350 56
037b7c2c 5b408728 20
037b7c40 5fe4506c 16
037b7c50 60876d60 32
037b7c70 5b408728 20
037b7c84 5fe4506c 16
037b7c94 00135de8 519112 Free
0383685c 5b408728 20
03836870 5fe4506c 16
03836880 608c55b4 96
....
There you find your free memory blocks which was an object which was already GCed. You can dump the surrounding objects (the output is sorted address wise) to find out if they are pinned or have other unusual properties.
You have 50MB of RAM as Free space. This is not good.
Having .NET allocating blocks of 16MB from process, we have a fragmentation issue indeed.
There are plenty of reasons to fragmentation to occure in .NET.
Have a look here and here.
In your case it is possibly a pinning. As 53304764 / 243010 makes 219.35 bytes per object - much lower then LOH objects.