we have a logstash and Elasticsearch-based solution to analyze network protocols, and I am trying to benchmark the whole solution. When I increase the number of Nodes for client/logstash and Elasticsearch, I am not seeing the expected scaling of the performance.
We have two protocols (protocol A and protocol B), there are separate logstash pipelines to process these protocols. To process each protocolweI have two logstash pipelines. i.e total 4 logstash pipelines, each protocol contains 2 logstash pipelines (pipeline 1 and pipeline2).
Pipeline Details:
I am decoding the packets related to the protocols and keeping them in two separate folders. Logstash pipeline-1 is monitoring a folder for new files (ndjson/ each file contains thousands of JSON objects). And for every event in pipeline 1 I am querying an elasticsearch index to get some metadata. I am using ruby and some other filters to process the events as shown below. The processed events are then ingested into an elaticsearch index. Each event contains a session id, and each event is tagged with the corresponding session ID before ingesting, and unique session ids are indexed in a separate index.
Logstash Pipeline 2 is reading from the index where I am storing the session ID’s. and for each session ID/tag, I am getting all the messages belonging to that session (search using tag). I am further stitching all the messages belonging to a particular session and ingesting it into a final index. While forming every session I am again querying an intermediate index for some metadata and after forming the session again I am updating the corresponding document in that index.
Setup Details:
4 machines each containing
16 cores
30 GB RAM
SSD NVMe
Note:
For all tests the ES JVM heap is configured to 15 GB (50% of total system memory). And logstash JVM heap is configured to 11 GB for each pipeline (a total of 22GB per machine as I am running two pipelines on one machine i.e pipeline 1 and pipeline 2).
we are doing a good amount of processing in ruby for each pipeline. Around 200 and 150 lines of ruby code for logstash pipeline 1 and pipeline 2 of Protocol A. Around 500 and 100 lines of ruby code for logstash pipeline 1 and pipeline 2 of Protocol B.
At any point in time, logstash pipeline 1 will be writing to the index other than the index that pipeline 2 is reading from. Once Pipeline 2 is done with reading all the data in the index it is working on it will exit. And I clean up all data from the index that pipeline 2 processed and restart pipeline 2 again, and this time it reads from the index that pipeline 1 was writing before. Also this time pipeline 1 writes to the index that pipeline 2 was reading previously. This toggling of the index is done every time after pipeline 2 restarts.
I ran several benchmarks by finetuning the workers, batch size and by modifying JVM Heap size for both Elasticsearch and Logstash but didn't see any improvements in the performance.
When I run the performance benchmarking for the solution with 2 nodes (1 for ES and 1 for logstash pipelines), I am getting 6k and 7k TPS for each protocol A and protocol B respectively. But when I benchmark both the protocols together with 4 nodes (2 Node ES Cluster and 1 Node for Protocol A, 1 Node for Protocol B pipelines), I am getting 9.5k combined TPS (4.5k TPS for Protocol A and 5 k for Protocol B).
Am I doing anything wrong which is affecting the performance of logstash/Elasticsearch? IsAre there any other parameres to fintune?
Any suggestions or feedback are appreciated.
Best Regards,
Rakhesh Kumbi
we are using Elasticsearch and Fluentd for Central logging platform. below is our Config details:
Elasticsearch Cluster:
Master Nodes: 64Gb Ram, 8 CPU, 9 instances
Data Nodes: 64Gb Ram, 8 CPU, 40 instances
Coordinator Nodes: 64Gb Ram, 8Cpu, 20 instances
Fluentd: at any given time we have around 1000+ fluentd instances writing logs to Elasticsearch coordinator nodes.
and on daily basis we create around 700-800 indices and which total to 4K shards on daily basis. and we keep maximum 40K shards on cluster.
we started facing performance issue on Fluentd side, where fluentd instances fails to write logs. common issues are :
1. read time out
2. request time out
3. {"time":"2021-07-02","level":"warn","message":"failed to flush the buffer. retry_time=9 next_retry_seconds=2021-07-02 07:23:08 265795215088800420057/274877906944000000000 +0000 chunk=\"5c61e5fa4909c276a58b2efd158b832d\" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error=\"could not push logs to Elasticsearch cluster ({:host=>\\\"logs-es-data.internal.tech\\\", :port=>9200, :scheme=>\\\"http\\\"}): [429] {\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"circuit_breaking_exception\\\",\\\"reason\\\":\\\"[parent] Data too large, data for [<http_request>] would be [32274168710/30gb], which is larger than the limit of [31621696716/29.4gb], real usage: [32268504992/30gb], new bytes reserved: [5663718/5.4mb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=17598408008/16.3gb, model_inference=0/0b, accounting=0/0b]\\\",\\\"bytes_wanted\\\":32274168710,\\\"bytes_limit\\\":31621696716,\\\"durability\\\":\\\"TRANSIENT\\\"}],\\\"type\\\":\\\"circuit_breaking_exception\\\",\\\"reason\\\":\\\"[parent] Data too large, data for [<http_request>] would be [32274168710/30gb], which is larger than the limit of [31621696716/29.4gb], real usage: [32268504992/30gb], new bytes reserved: [5663718/5.4mb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=17598408008/16.3gb, model_inference=0/0b, accounting=0/0b]\\\",\\\"bytes_wanted\\\":32274168710,\\\"bytes_limit\\\":31621696716,\\\"durability\\\":\\\"TRANSIENT\\\"},\\\"status\\\":429}\"","worker_id":0}
looking for guidance on this, how we can optimise our Logs cluster?
Well, by the looks of it, you have exhausted your parent circuit breaker limit of 95% of Heap Memory.
The error you mentioned has been mentioned in the elasticsearch docs -
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/current/fix-common-cluster-issues.html#diagnose-circuit-breaker-errors
. The page also refers to a few steps you can take to Reduce JVM memory pressure, which can be helpful to reduce this error.
You can also try increasing this limit to 98%, using the dynamic command -
PUT /_cluster/settings
{
"persistent" : {
"indices.breaker.total.limit" : "98%"
}
}
But I would suggest this be performance tested before applying in production.
Since your request is 30GB, which is a bit too much, for a more reliable solution, I would suggest increasing your log scrapers frequency, so that it makes more frequent posts to ES with smaller-sized data blocks.
we are using ES-HADOOP plugin to push data into Elasticsearch cluster from Hadoop HBASE table. below are the cluster details.
elasticsearch version: 2.3.5
data nodes: 3
master nodes: 3
client node: 1
the data nodes are master nodes as well.
data/master nodes heap: 20GB
client nodes heap: 3GB
Number of Primary Shards per index: 5
Number of Replica Shards per index: 1
when we execute jobs on Spark and on stages where we push data from Hadoop to Elasticsearch after some time we start getting ElasticSearch Bailing Out.
we suspect that number of concurrent connections which Elasticsearch can process for Bulk API is exceeding by the Spark Executors due to which post maximum numbers of connections Elasticsearch start rejecting the write requests.
How we can identify that how much concurrent bulk API connection can ElasticSearch Client node can process and successfully write the data and what should be the maximum number of documents per BULK API REQUEST?
What parameters we should look into optimise the ElasticSearch cluster for write operations where we need to index 80-90 GB data in a hour?
We have an elastic search cluster of 3 nodes of the following configurations
#Cpu Cores Memory(GB) Disk(GB) IO Performance
36 244.0 48000 very high
The machines are in 3 different zones namely eu-west-1c,eu-west-1a,eu-west-1b.
Each elastic search instance is being allocated 30GB of heap space.
we are using the above cluster for running aggregations only. The cluster has replication factor of 1 and all the string fields are not analyzed , doc_values is true for all the fields.
We are pumping data into this cluster running 6 instances of logstash in parallel ( having a batch size of 1000)
When more instances of logstash are started one by one the nodes of the ElasticSearch cluster starts throwing out of memory error.
What could be the possible optimizations to speed up bulk indexing rate on the cluster?= Will presence of nodes of cluster in the same zone increase bulk indexing? Will adding more nodes in the cluster help ?
Couple of steps taken so far
Increase the bulk queue size from 50 to 1000
Increase refresh interval from 1 seconds to 2 minutes
Changed segments merge throttling to none (
https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing- performance.html)
We cannot set the replication factor to 0 due to inconsistency involved if one of the nodes goes down.
Currently we are using spark with 1 minute batch interval to process the data. The data flow is like HTTP endpoint -> Spring XD -> kafka -> Spark Streaming -> HBASE. Format is JSON. We are running the spark jobs in a environment which has 6 nodemanagers each with 16 CPU cores and 110 GB of RAM. For caching metedata scala's triemap is used,so the cache will be per executor. Results on Spark with below settings:
Kafka partitions - 45
Spark executors - 3
Cores per executor -15
Number of JSON records 455 000 received over a time of 10 minutes.
Spark processed the records in 12 minutes. And each executor is able to process about 350-400 records per sec. Json parsing, validations, other stuffs are done before loading into HBASE.
With almost the same code with modifications for flink I ran the code with flink streaming deployed in YARN cluster. Results on Flink with below settings:
Kafka partitions - 45
Number of Task Managers for running the job - 3
Slots per TM - 15
Parallelism - 45
Number of JSON records 455 000 received over a time of 10 minutes.
But flink takes almost 50 minutes to process the records. With each TM process 30-40 records per second.
What am i missing here? Are there any other parameters/Configurations apart from the mentioned one impacts performance? The Job flow is DataStream -> Map -> custom functions. How could I improve the performance of flink here?