How to choose optimal logstash pipleline batch size and delay? (Logstash 6.4.3) - elasticsearch

Question

Introduction
We have a logstash that is receiving our logs from java microservices, and lately the machine has been at 100% utilization.
I noticed that very low values were used for pipeline batch size, workers, and delay as well as ram.
My feeling was that I could improve performance by increasing the batch size into the thousands, increasing the delay into the seconds, and increasing the ram.
It seems to have worked and we have gone from a logstash that was crashing at 100% continously (or close to it) to being at (or below) 70%. This is a virtual server running in vmware with only 1 core assigned so resources are a bit limited.
Question
How do I optimize further? (without messing with the microservices or limiting the number of incoming message)?
How do I find the optimal values for delay and batch size?
Also, even though we have 1 core, I have the feeling that having more than 1 worker helps but I'm not sure about that (due to IO delays)
Current config
ELK (Elastic, Logstash, Kibana) 6.4
logstash.yml contains
pipeline:
batch:
size: 2048
delay: 5000
pipeline.workers: 4
Elastic jvm.options
-Xms4g
-Xmx10g
Logstash jvm.options
-Xms4g
-Xmx10g
Logstash config:
input {
tcp {
port => 8999
codec => json
}
}
filter {
geoip {
source => "req.xForwardedFor"
}
}
filter {
kv {
include_keys => [ "freeTextSearch", "entityId","businessId"]
recursive => "true"
field_split => ","
}
}
filter {
mutate {
split => { "req.user" => "," }
split => { "req.application" => "," }
split => { "req.organization" => "," }
split => { "app.profiles" => "," }
copy => { "app.name" => "appLicationName" }
}
}
filter {
fingerprint {
target => "[#metadata][uuid]"
method => "UUID"
}
}
filter {
if [app]
{
ruby
{ init => '
BODY_PATH = "[app]"
BODY_STRING = "[name]"
'
code => '
body_val = event.get(BODY_PATH)
if body_val.is_a?(String)
event.set(BODY_PATH, {BODY_STRING => body_val,"[olderApp]" => "true"})
end
'
}
}
}
output {
stdout {
codec => rubydebug {
metadata => true
}
}
if [stackTrace] {
email {
address => 'smtp.internal.email'
to => 'Warnings<warning#server.internal.org>'
from => 'Warnings<warning#server.internal.org>'
subject => '%{message}'
template_file => "C:\logstash\emailtemplate.mustache"
port => 25
}
}
elasticsearch {
hosts => ["localhost:8231"]
sniffing => true
manage_template => false
index => "sg-logs"
document_id => "%{[#metadata][uuid]}"
}
}
Update
I switched to the persistent queue, which has improved things quite a bit in terms of performance. I ran the scripts that used to freeze our logtash and it seems to not be breaking, though it took quite a bit of work.
Switched to pipeline.yml
I switched to pipeline.yml since I noticed that the queue settings were not working. I also had to pass the YML through a validator.
---
-
path.config: "../configsg/"
pipeline.batch.size: 1000
pipeline.id: persisted-queue-pipeline
pipeline.workers: 2
queue.type: persisted
queue.max_bytes: 2000mb
queue.drain: true
Modified the bat file to clean the data/queue folder
I noticed logstash wasn't processing correctly when there was leftover data inside data/queue folder. I added a bat file to clean/move this data during logstash restarts etc. I need to think about how to handle this in the future.
Folder: logstash-6.4.3\data\queue
Here is my bat file that is called by a windows service during starts/restarts.
echo Date format = %date%
echo dd = %date:~0,2%
echo mm = %date:~3,2%
echo yyyy = %date:~6,8%
echo.
echo Time format = %time%
echo hh = %time:~0,2%
echo mm = %time:~3,2%
echo ss = %time:~6,2%
cd ..
cd data/queue
move ./persisted-queue-pipeline ../persist-queue-backup-%date:~0,2%_%date:~3,2%_%date:~6,8%-%time:~0,2%_%time:~3,2%_%time:~6,2%.txt
cd ../../bin
logstash.bat

Answer 1

Here some tips from Logstash team about optimization: link
I would also suggest taking a look at multi-pipeline cases. From your config, it sounds to me filter cases may causing the backpressure. It seems if you can divide your input (by port), you can set multiple pipeline to handle the backpressure cases.