elasticsearch 1.7.2 on CentOS, 8GB RAM, multi node cluster.
We are reviewing: https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html
and have a few questions.
1) Other than the breakers, how do we see approaching max-out of fielddata RAM ? (The stats in /_stats/fielddata?fields=* , like memory_size, don't show 'available_memory')
2) When we run /_stats/fielddata?fields=*, we see "primaries" and "total", which is fine. But what is the total a total of? It appears to be memory_size * number of nodes, (but that doesn't make sense to us).
3) We moved from nodes with 4GB RAM to nodes with 8GB RAM, yet we feel like the avail RAM (on stock elasticsearch.yml) for fielddata did not double. Is that a fair conclusion?
The essential thing we want to monitor is: How close are we to maxing RAM... but we don't see what to measure.
First of all, you are not maxing RAM, but HEAP. ES is using both Heap memory (from the JVM) and, also, RAM (as file data cache).
To see how much heap your nodes are using, this is the command to use: GET /_nodes/stats/jvm, which will output something like
"jvm": {
"timestamp": 1447790529519,
"uptime_in_millis": 84853,
"mem": {
"heap_used_in_bytes": 311374768,
"heap_used_percent": 14,
"heap_committed_in_bytes": 2112618496,
You need 100 * heap_used_in_bytes / heap_committed_in_bytes and resulting a percentage of usage. Or simply ysing heap_used_percent. For the actual RAM usage, use your OS reports for that or GET /_nodes/stats/os.
For fielddata you can get statistics per node using GET /_nodes/stats/indices/fielddata
Related
How can I determine the heap size required for 1 GB logs having 1 day retention period?
if I take the machine with 32 GB heap size (64 GB RAM) how many GB logs I can keep in this for 1 day?
It depends on various factors like the number of indexing requests, search requests, cache utilization, size of search and indexing requests, number of shards/segments etc, also heap size should follow the sawtooth pattern, and instead of guessing it, you should start measuring it.
The good thing is that you can starting right, by assigning 50% of RAM as ES Heap size which is not crossing 32 GB.
When querying my cluster, I noticed these stats for one of my nodes in the cluster. Am new to Elastic and would like the community's health in understanding the meaning of these and if I need to take any corrective measures?
Does the Heap used look on the higher side and if yes, how would I rectify it? Also any comments on the System Memory Used would be helpful - it feels like its on the really high side as well.
These are the JVM level stats
JVM
Version OpenJDK 64-Bit Server VM (1.8.0_171)
Process ID 13735
Heap Used % 64%
Heap Used/Max 22 GB / 34.2 GB
GC Collections (Old/Young) 1 / 46,372
Threads (Peak/Max) 163 / 147
This is the OS Level stats
Operating System
System Memory Used % 90%
System Memory Used 59.4 GB / 65.8 GB
Allocated Processors 16
Available Processors 16
OS Name Linux
OS Architecture amd64
As You state that you are new to Elasticsearch I must say you go through cluster as well as cat API you can find documentation at clusert API and cat API
This will help you understand more in depth.
I'm using the new TensorFlow profiler to profile memory usage in my neural net, which I'm running on a Titan X GPU with 12GB RAM. Here's some example output when I profile my main training loop:
==================Model Analysis Report======================
node name | requested bytes | ...
Conv2DBackpropInput 10227.69MB (100.00%, 35.34%), ...
Conv2D 9679.95MB (64.66%, 33.45%), ...
Conv2DBackpropFilter 8073.89MB (31.21%, 27.90%), ...
Obviously this adds up to more than 12GB, so some of these matrices must be in main memory while others are on the GPU. I'd love to see a detailed breakdown of what variables are where at a given step. Is it possible to get more detailed information on where various parameters are stored (main or GPU memory), either with the profiler or otherwise?
"Requested bytes" shows a sum over all memory allocations, but that memory can be allocated and de-allocated. So just because "requested bytes" exceeds GPU RAM doesn't necessarily mean that memory is being transferred to CPU.
In particular, for a feedforward neural network, TF will normally keep around the forward activations, to make backprop efficient, but doesn't need to keep the intermediate backprop activations, i.e. dL/dh at each layer, so it can just throw away these intermediates after it's done with these. So I think in this case what you care about is the memory used by Conv2D, which is less than 12 GB.
You can also use the timeline to verify that total memory usage never exceeds 12 GB.
I want to calculate the actual amount of mflop/s/core using the following information:
I have measured actual amount of memory bandwidth of each core in 1 node which is 4371 MB/s.
I have also measured mflop/s/core on one node if I use only one core on a node(in this case the whole memory of node would be available for that core), the result is 2094.45. So I measured the memory bandwidth which was available for that core = 10812.3 MB/s
So now I want to calculate the actual mflop/s/core when the core has it's real memory bandwidth (4371MB/s).
Do you think it would be correct if I calculate it like this:
actual mflop/s/core= (mflop/s/core * actual memory bw) / used memory bandwidth
Any help would be appreciated.
We use ES to store around 2.5TB of data. We have 12 primary shards and 2 replica shards.
We are currently load testing ES and I read the following article
https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html
This article states 2 important things. First allocate 50% of Memory to Lucene and Second Don't cross 30.5GB limit for heap space.
I don't clearly understand the 30.5GB limit. I understand that if I am to set 40GB over 30.5 GB i will loose more than i gain(because of compressed pointers) but say if i have hardware of around 250GB RAM what are the reasons that i should only allocate 30.5GB and not 120GB for heap. Won't i start seeing gains after 70-80GB heap setting over 30.5 GB heap. Can somebody list down all the reasons?