I have 3 pods in kubernetes written in springboot. The max memory allocated is 1.5 GB. The container during some load scaled to 3 from inital 2. Now the memory usage in each POD is 1GB. But even during non peak hours the scale down is not happening.The memory usage is consistent in 3 pods with almost 1GB usage.
The application uses Java 11 and the GC of Java 11 seems to be G1 which might not be doing a full memory freeing. Could memory leak be a reason its not going down or because of G1 GC.
But if its memory leak it shouldn't be stable and keeps on increasing?
Can anyone tell me what exactly could be happening in these scenario?
Related
There seems to be a few issues raised about geoserver's memory usage - I'm wondering if someone can speculate as to why my memory use is so high.
I'm currently running geoserver 2.11.1 under tomcat7 with a -Xms set to 8gb and -Xmx set to 12Gb. I have 45,000 layers containing small shapefiles (~3Mb each) across the same number of stores and around 60 workspaces.
Memory use, using the rest of the default values from tomcat and geoserver, immediately reaches 8Gb and I can't figure out why this would be the case.
Thoughts include:
1) Garbage collection is inefficient
2) There is a memory leak
3) There is not enough memory on the server
Hopefully someone can shine some light on why memory use is so high - any help is appreciated!
The Xms is the startup memory size, so it is expected that your application takes this amount of memory. Now let's not forget that it reserves this memory, it doesn't necessarily use it right away.
I'm running neo4j 3.2.1 on Linux machine of 16GB RAM, yet I'm having issues ith the heap memory showing an error everytime.
In the documentation, we have:
Actual OS allocation = available RAM - (page cache + heap size)
Does it mean that if we configure it for my machine (for example 16g for the heap and 16g for page cache) then the Allocation would be 0, inducing the problem?
Can anyone tell me how to proceed with the best configuration to explore more of the capacity of the machine without having to face that heap error again?
In your example, you are trying to give all the RAM to the page cache, and also all the RAM to the heap. That is never possible. The available RAM has to be divided between the OS, the page cache, and the heap.
The performance documentation shows how to divide up the RAM.
As a first pass, you could try this allocation (given your 16 GB of RAM):
7 GB for the page cache
8 GB for the heap
That will leave (16GB - (7GB + 8GB)), or 1 GB for the OS.
But you should read the documentation to fine tune your allocations.
I'm getting this memory error on my reducers:
Container is running beyond physical memory limits. Current usage: 6.1 GB of 6 GB physical memory used; 10.8 GB of 30 GB virtual memory used.
So yeah, there's a physical memory issue what can be resolved by increasing mapreduce.reduce.memory.mb BUT I don't understand why it happens.
The more data is getting into the pipe, the more likely it is for this memory issue to occur. The thing is that most of my reducers (about 90%) pass and memory was supposed to be freed as reducers pass because data should have been written to the disk already.
What am I missing here?
In YARN by default, containers are pre allocated with memory. In this case all reducer containers will have 6 GB memory regardless of free memory across cluster. The JVM of these containers will not be re-used or shared across containers(not at least in Hadoop2) , that means If the memory on one reducer goes beyond its memory 6 GB limit, it will not grab resources from other free containers (if that's your concern)
Now , why only few reducers are always going above its memory(given 90% passed) hints a possible SKEW in data , which means this reducer might be processing more input groups or more keys than others.
We currently have a ArangoDB cluster running on version 3.0.10 for a POC with about 81 GB stored on disk and Main memory consumption of about 98 GB distributed across 5 Primary DB servers. There are about 200 million vertices and 350 million Edges. There are 3 Edge collections and 3 document collections, most of the memory(80%) is consumed due to the presence of the edges
I'm exploring methods to decrease the main memory consumption. I'm wondering if there are any methods to compress/serialize the data so that less amount of main memory is utilized.
The reason for decreasing memory is to reduce infrastructure costs, I'm willing to trade-off on speed for my use case.
Please can you let me know, if there any Methods to reduce main memory consumption for ArangoDB
It took us a while to find out that our original recommendation to set vm.overcommit_memory to 2 this is not good in all situations.
It seems that there is an issue with the bundled jemalloc memory allocator in ArangoDB with some environments.
With an vm.overcommit_memory kernel settings value of 2, the allocator had a problem of splitting existing memory mappings, which made the number of memory mappings of an arangod process grow over time. This could have led to the kernel refusing to hand out more memory to the arangod process, even if physical memory was still available. The kernel will only grant up to vm.max_map_count memory mappings to each process, which defaults to 65530 on many Linux environments.
Another issue when running jemalloc with vm.overcommit_memory set to 2 is that for some workloads the amount of memory that the Linux kernel tracks as "committed memory" also grows over time and does not decrease. So eventually the ArangoDB daemon process (arangod) may not get any more memory simply because it reaches the configured overcommit limit (physical RAM * overcommit_ratio + swap space).
So the solution here is to modify the value of vm.overcommit_memory from 2 to either 1 or 0. This will fix both of these problems.
We are still observing ever-increasing virtual memory consumption when using jemalloc with any overcommit setting, but in practice this should not cause problems.
So when adjusting the value of vm.overcommit_memory from 2 to either 0 or 1 (0 is the Linux kernel default btw.) this should improve the situation.
Another way to address the problem, which however requires compilation of ArangoDB from source, is to compile a build without jemalloc (-DUSE_JEMALLOC=Off when cmaking). I am just listing this as an alternative here for completeness. With the system's libc allocator you should see quite stable memory usage. We also tried another allocator, precisely the one from libmusl, and this also shows quite stable memory usage over time. The main problem here which makes exchanging the allocator a non-trivial issue is that jemalloc has very nice performance characteristics otherwise.
(quoting Jan Steemann as can be found on github)
Several new additions to the rocksdb storage engine were made meanwhile. We demonstrate how memory management works in rocksdb.
Many Options of the rocksdb storage engine are exposed to the outside via options.
Discussions and a research of a user led to two more options to be exposed for configuration with ArangoDB 3.7:
--rocksdb.cache-index-and-filter-blocks-with-high-priority
--rocksdb.pin-l0-filter-and-index-blocks-in-cache
--rocksdb.pin-top-level-index-and-filter
Redis support 3 memory allocator: libc, jemalloc, tcmalloc. When i do memory usage test, i find that mem_fragmentation_ratio in INFO MEMORY could be less than 1 with libc allocator. With jemalloc or tcmalloc, this value is greater or equal than 1 as it should be.
Could anyone explain why mem_fragmentation_ratio is less than 1 with libc?
Redis version:2.6.12. CentOS 6
Update:
I forgot to mention that one possible reason is that swap happens and mem_fragmentation_ratio will be < 1.
But when i do my test, i adjust swapiness, even turn swap off. The result is the same. And my redis instance actually do not cost too much memory.
Generally, you will have less fragmentation with jemalloc or tcmalloc, than with libc malloc. This is due to 4 factors:
more granular allocation classes for jemalloc and tcmalloc. It reduces internal fragmentation, especially when Redis has to allocate a lot of very small objects.
better algorithms and data structures to prevent external fragmentation (especially for jemalloc). Obviously, the gain depends on your long term memory allocation patterns.
support of "malloc size". Some allocators offer an API to return the size of allocated memory. With glibc (Linux), malloc does not have this capability, so it is emulated by explicitly adding an extra prefix to each allocated memory block. It increases internal fragmentation. With jemalloc and tcmalloc (or with the BSD libc malloc), there is no such
overhead.
jemalloc (and tcmalloc with some setting changes) can be more aggressive than glibc to release memory to the OS - but again, it depends on the allocation patterns.
Now, how is it possible to get inconsistent values for mem_fragmentation_ratio?
As stated in the INFO documentation, the mem_fragmentation_ratio value is calculated as the ratio between memory resident set size of the process (RSS, measured by the OS), and the total number of bytes allocated by Redis using the allocator.
Now, if more memory is allocated with libc (compared to jemalloc,tcmalloc), or if more memory is used by some other processes on your system during your benchmarks, Redis memory may be swapped out by the OS. It will reduce the RSS (since a part of Redis memory will not be in main memory anymore). The resulting fragmentation ratio will be less than 1.
In other words, this ratio is only relevant if you are sure Redis memory has not been swapped out by the OS (if it is not the case, you will have performance issues anyway).
Other than swap, I know 2 ways to make "memory fragmentation ratio" to be less than 1:
Have a redis instance with little or no data, but thousands of idling client connections. From my testing, it looks like redis will have to allocate about 20 KB of memory for each client connections, but most of it won't actually be used (i.e. won't appear in RSS) until later.
Have a master-slave setup with let's say 8 GB of repl-backlog-size. The 8 GB will be allocated as soon as the replication starts (on master only for version <4.0, on both master and slave otherwise), but the memory will only be used as we start writing to the master. So the ratio will be way below 1 initially, and then get closer and closer to 1 as the replication backlog get filled.