Logstash+Elasticsearch throughput - elasticsearch

Question

we're trying to process 5K msgs/sec with 2 identical machines, but seems like we max out logstash or elasticsearch.
Each has:
64Gb RAM, ≈3Ghz Xeon CPU
Logstash 1.5 installed
Elasticsearch 1.7.8 installed in cluster mode with second machine.
Logstash is configured to receive messages from 16-node kafka cluster and send it to Elasticsearch.
Data is CSV, contains 22 fields. Is that a normal throughput?
Here's the config:
input{
kafka {
type => "api"
zk_connect => "node1:2181,node2:2181,node3:2181"
codec => "plain"
topic_id => "api_events"
consumer_threads => 8
queue_size => 10000
rebalance_backoff_ms => 10000
rebalance_max_retries => 10
}
}
filters{
csv {
separator => "::"
columns => [
"hostname",
"status",
"body_bytes_sent",
"request_time",
"http_x_forwarded_for",
"uri",
"arg_key",
"http_user_agent",
"http_deviceid",
"http_country_code",
"http_language_code",
"http_platform",
"http_versioncode",
"request_method",
"http_x_forwarded_proto",
"upstream_cache_status",
"upstream_response_time",
"upstream_header_time",
"upstream_status",
"bytes_sent",
"time_local",
"upstream_addr"
]
remove_field => [
"message"
]
}
mutate {
convert => {
"body_bytes_sent" => "integer"
"request_time" => "float"
"upstream_response_time" => "float"
"upstream_header_time" => "float"
"bytes_sent" => "integer"
}
}
}
}
output{
elasticsearch {
cluster => "MyCluster"
protocol => "node"
index => "api-%{+YYYY.MM.dd}"
host => "elasticnode1"
flush_size => 50000
workers => 4
}
}

Answer 1

I'm surprised this question has not been answered. The question has been asked with rather old versions of logstash There have been multiple improvements and refactoring in Logstash since so some of the parameters will be different.
5k messages a second, at least on the face of it, sounds pretty low to achieve but of course it depends on quite a few things which are not stated. For example, wow big is each message? How many partitions is it listening from? Is each partition in the input actually receiving messages at high throughputs or is only the aggregate throughput high?
I would suggest starting with a small batch size (say 500) with 1 worker and slowly increasing the batch size till you see no improvement and then increasing the workers to make use of the cores on the machine. It is possible you're not getting each batch full enough per request per worker. The following article shows how to profile and measure how the in flight requests are doing with real arriving data:
https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html
Of course the other side of this is the Elasticsearch side. It is worth measuring that Elasticsearch is not lagging in processing all he concurrent requests. How many Elasticsearch nodes are there and what are the numbers of Client/Data nodes? There are a number of things to look out for on Elasticsearch as well when doing "heavy" indexing:
https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html
I hope this helps you and others looking to do this today.