We are starting designing a cluster and come up with the following optimal configuration . Please suggest if there is any scope for improvement or save some budget if it is over optimized.
100 fields - 1 MB per field(including inverted index) - 125 MB( to be safer side) per document
Total - 4M documents corresponds to 500 GB - 25GB per shard - 20 shards - total 40 shards including ( r = 1) - we had seen somewhere a shard size of 25GB works well in most of the scenarios.
Also it seems heap max 32 GB per each shard RAM) ( jvm uses compressed pointers ) works well - which translates to 64 GB (rest 50% for FS cache) .So , considering 256 GB RAM - this translates 2 shards per machine (128GB) - this translates 20 Data nodes per cluster(2 shards per each data node ) , 3 master nodes (HA ) , 1 coordinating node
Please add your recommendations
Related
I have 8 node cluster(3Master+3Data+2Coordinate) in each data node the heap size is 10gb and disk space is 441gb each total 1.2TB
in each day i have 32.73GB data in each day 26 shards created for 11 indices.So lets suppose the retention period is 30 days.In 30th day the data on cluster would be 982GB and total shards would be 780 each node gets 260 shards.So the average shard size would be 260mb(approx).i read this documentation that a node of 30gb heap size can handle 600 shards.So the question is Can heap size of 10gb can handle 260 shards ?.
This article which you read can be considered a good general recommendations but there are various factors which can affect it, like size of indices, size of shards, size of documents, type of disk, current load on the system and so on, in the same document you can notice the shard size recommendation is between 10 to 50 GB, while you have very small shard size(260 MB as you mentioned), so based on this, I can say 10GB heap can easily handle 260 shards in your case, although you should benchmark your cluster and read more about how ES internally stores the data and searches them so that its easy for you to fine-tune it.
Apache NiFi is spiking the CPU to 100% and the specific node gets disconnected from the cluster.
We have 3 nodes cluster,
each with 8 core, and 64 GB RDM and 40 GB memory allocated to the JVM
In Merge Content when processing 2 million + small flow small flow files ( size of all files around 300 MB), NiFi node cpu reaches 100% and node gets disconnected and data never gets processed,
I am using defragment strategy in merge content.
I tried changing the nifi.queue.swap.threshold to 1,000,00 but it does not help.
Any hints on how to deal with this ?
I am trying to implement a hot-warm architecture in order to configure a 6 month data retention (36 TB, 200 GB per day) policy in which data has to be deleted from an index 180 days after it has been indexed.
A hot index older than 40 days will enter the warm phase.
I would like to know How many nodes per phase should my cluster have? How many shard/ replicas should I create? What is the optimal average size of a shard for the best searching performance? What is the Capacity of RAM / Storage / CPU per node ?
Any help is appreciated.
I see that indexing performance degraded over a period of time in Elasticsearch. I see that the mem usage has slowly increased over a period of time until it became 100%. At this state I cannot index any more data. I have default shard settings - 5 primary and 1 replica. My index is time based with index created every hour to store coral service logs of various teams. An index size corresponds to about 3GB with 5 shards and with replica it is about 6GB. With a single shard and 0 replicas it comes to about 1.7 GB.
I am using ec2's i2.2x large hosts which offer 1.6TB space and 61GB RAM and 8 cores.
I have set the heap size to 30GB.
Following is node statistics:
https://jpst.it/1eznd
Could you please help in fixing this? My whole cluster came down that I had to delete all the indices.
I'm experiencing search performance degradation when index size is growing. Eg: For a given query, i'm getting 3500 RPS (Request per second) with 1M documents on index, whereas with 6M documents on index is giving me 1200 RPS. Each document size on index is average of 500 bytes. Tried with following different cluster configs, and the behavior is same in both cases.
Elastic Search version: 1.2.1
Cluster 1 configuration: 4 node cluster
2 Routers Nodes: VMs with 4 Core CPU, 4 GB Ram, 20GB disk
2 Data Nodes: Physicals with 32 Core CPU, 198GB Ram (24G allocated to ES process), 250 GB disk
Number of shards for index : 3 and 1 replica
Cluster 2 configuration: 7 node cluster
2 Routers Nodes: VMs with 4 Core CPU, 4 GB Ram, 20GB disk
2 Data Nodes: VMs with 8 Core CPU, 48GB Ram (24G allocated to ES process), 250 GB local storage
Number of shards for index : 3 and 1 replica
In addition to search query performance issue, i observed that the Data nodes CPU is always spiking to 100% on all core.
Any inputs would be greatly appreciated.
Thanks
Gopi