Logstash stops processing logs after some hours. When logs stops processing logstash service consumes high amount of CPU performance (about 25 cores of 32 total).
When logstash service works normal it consumes about 4-5 cores total.
Pipeline generates about 50k events per minute.
Logstash Conf (non default):
pipeline.workers: 15
pipeline.batch.size: 100
JVM CONF:
-Xms15g
-Xmx15g
input {
tcp {
port => 5044
type => syslog
}
udp {
port => 5044
type => syslog
}
}
filter {
if [type] == "syslog" {
grok {
match => [ "message", "%{SYSLOG5424PRI}%{NOTSPACE:syslog_timestamp} %{NOTSPACE:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ]
}
kv {
id => "logs_kv"
source => "syslog_message"
trim_key => " "
trim_value => " "
value_split => "="
field_split => " "
}
mutate {
remove_field => [ "syslog_message", "syslog_timestamp" ]
}
#now check if source IP is a private IP, if so, tag it
cidr {
address => [ "%{srcip}" ]
add_tag => [ "src_internalIP" ]
network => [ "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16" ]
}
# don't run geoip if it's internalIP, otherwise find the GEOIP location
if "src_internalIP" not in [tags] {
geoip {
add_tag => [ "src_geoip" ]
source => "srcip"
database => "/usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-City.mmdb"
}
geoip {
source => "srcip"
database => "/usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-ASN.mmdb"
}
}
else {
#check DST IP now. If it is a private IP, tag it
cidr {
add_tag => [ "dst_internalIP" ]
address => [ "%{dstip}" ]
network => [ "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16" ]
}
# don't run geoip if it's internalIP, otherwise find the GEOIP location
if "dst_internalIP" not in [tags] {
geoip {
add_tag => [ "dst_geoip" ]
source => "dstip"
database => "/usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-City.mmdb"
}
geoip {
source => "dstip"
database => "/usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-ASN.mmdb"
}
}
}
}
}
output {
if [type] == "syslog" {
elasticsearch {hosts => ["127.0.0.1:9200" ]
index => "sysl-%{syslog_hostname}-%{+YYYY.MM.dd}"
}
#stdout { codec => rubydebug }
}
}
When logstash stops processing i dont see any errors in log file (log level - trace). Only see these messages:
[2019-04-19T00:00:12,004][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2019-04-19T00:00:17,011][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2019-04-19T00:00:17,012][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2019-04-19T00:00:22,015][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2019-04-19T00:00:22,015][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2019-04-19T00:00:27,023][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2019-04-19T00:00:27,024][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2019-04-19T00:00:32,030][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2019-04-19T00:00:32,030][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
events format:
[2019-04-22T13:04:27,190][DEBUG][logstash.pipeline ] filter received {"event"=>{"type"=>"syslog", "#version"=>"1", "#timestamp"=>2019-04-22T10:04:27.159Z, "port"=>50892, "message"=>"<30>2019:04:22-13:05:08 msk ulogd[18998]: id=\"2002\" severity=\"info\" sys=\"SecureNet\" sub=\"packetfilter\" name=\"Packet accepted\" action=\"accept\" fwrule=\"6\" initf=\"eth2\" outitf=\"eth1\" srcmac=\"70:79:b3:ab:e0:e8\" dstmac=\"00:1a:8c:f0:89:02\" srcip=\"10.0.134.138\" dstip=\"10.0.131.134\" proto=\"17\" length=\"66\" tos=\"0x00\" prec=\"0x00\" ttl=\"126\" srcport=\"63936\" dstport=\"53\" ", "host"=>"10.0.130.235"}}
Help me please debug this problem.
According to the internet, the ParNew garbage collector is "stop the world". If it takes 5 seconds to resume, and you're getting GC every 5 seconds, you'll get no throughput from logstash as it's always blocked.
Solved.
Problem was in kv filter, which stopped logstash when trying to parse non-structured data going trough pipeline.
Related
I'm forwarding application logs to elasticsearch, while performing some grok filters before.
The application has a timestamp field and there's the timestamp field of logstash itself.
We regularly check the difference between those timestamp, and on many cases the delay is very big, meaning the log took very long time to be shipped to elasticsearch.
I'm wondering how can I isolate the issue to know if the delay is coming from logstash or elasticsearch.
Example logstash scrape config:
input {
file {
path => "/app/app-core/_logs/app-core.log"
codec => multiline {
pattern => "(^[a-zA-Z.]+(?:Error|Exception).+)|(^\s+at .+)|(^\s+... \d+ more)|(^\t+)|(^\s*Caused by:.+)"
what => "previous"
}
}
}
filter {
if "multiline" not in [tags]{
json {
source => "message"
remove_field => ["[request][body]","[response][body][response][items]"]
}
}
else {
grok {
pattern_definitions => { APPJSON => "{.*}" }
match => { "message" => "%{APPJSON:appjson} %{GREEDYDATA:stack_trace}"}
remove_field => ["message"]
}
json {
source => "appjson"
remove_field => ["appjson"]
}
}
}
output {
elasticsearch {
hosts => ["elasticsearch-logs.internal.app.io:9200"]
index => "logstash-core-%{+YYYY.MM.dd}"
document_type => "logs"
}
}
We tried adjusting the number of workers and batch size, no value we tried reduced the delay:
pipeline.workers: 9
pipeline.output.workers: 9
pipeline.batch.size: 600
pipeline.batch.delay: 5
Nothing was done on the elasticsearch side because I think the issue is with logstash, but I'm not sure.
I am trying to configure logstash to aggregate similar syslog based on a message field and in a specific timestamp.
To make my case clear, this is an example of what I would like to do.
example: I have those junk syslog coming through my logstash
timestamp. message
13:54:24. hello
13:54:35. hello
What I would like to do is have a condition that check if the message are the same and those message occurs in a specific timespan (for example 10min) I would like to aggregate them into one row, and increase the count
the output I am expecting to see is as follow
timestamp. message. count
13.54.35. hello. 2
I know and I saw that there is the opportunity to aggregate the fields, but I was wondering if there is a chance to do this aggregation based on a specific time range
If anyone can help me I would be extremely grateful as I am new to logstash and I have the problem that in my server I am receiving tons of junk syslog and I would like to reduce that amount.
So far I did some cleaning with this configuration
input {
syslog {
port => 514
}
}
filter {
prune {
whitelist_names =>["timestamp","message","newfield"]
}
mutate {
add_field => {"newfield" => "%{#timestamp}%{message}"}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash_index"
}
stdout {
codec => rubydebug
}
}
Now I just need to do the aggregation.
Thank you so much for your help guys
EDIT:
Following the documentation, I put in place this configuration:
input {
syslog {
port => 514
}
}
filter {
prune {
whitelist_names =>["timestamp","message","newfield"]
}
mutate {
add_field => {"newfield" => "%{#timestamp}%{message}"}
}
if [message] =~ "MESSAGE FROM" {
aggregate {
task_id => "%{message}"
code => "map['message'] ||= 0; map['message'] += 1;"
push_map_as_event_on_timeout => true
timeout_task_id_field => "message"
timeout => 60
inactivity_timeout => 50
timeout_tags => ['_aggregatetimeout']
timeout_code => "event.set('count_message', event.get('message') > 1)"
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash_index"
}
stdout {
codec => rubydebug
}
}
I don't get any error but the output is not what I am expecting.
The actual output is that it create a tag field (Good) passing an array with _aggregationtimeout and _aggregationexception
{
"message" => "<88>MESSAGE FROM\r\n",
"tags" => [
[0] "_aggregatetimeout",
[1] "_aggregateexception"
],
"#timestamp" => 2021-07-23T12:10:45.646Z,
"#version" => "1"
}
I have logstash filebeat and elasticsearch running on one node.
I'm trying to get logstash to identify logs labeled as "syslog" and dump them in an index named "syslog", but it appears to not see the label as they are all going into the "uncategorized" index (my catch all default index)
Here is my beats config
/etc/filebeat/filebeat.yml
filebeat:
prospectors:
-
paths:
- /var/log/messages
fields:
type: syslog
output:
logstash:
hosts: ["localhost:9901"]
Here is my logstash config file
/etc/logstash/conf.d/logstash_server_syslog.conf
input {
beats {
port => "9901"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{#timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
if [type] == "syslog" {
elasticsearch {
hosts => ["10.0.0.167:9200", "10.0.0.168:9200"]
index => "syslog"
}
} else {
elasticsearch {
hosts => ["10.0.0.167:9200", "10.0.0.168:9200"]
index => "uncategorized"
}
}
}
Looking at the output (with a stdout{} stanza) would confirm this, but I'm guessing that you missed this part of the doc:
By default, the fields that you specify [in the 'fields' config'] will be grouped under a
fields sub-dictionary in the output document. To store the custom
fields as top-level fields, set the fields_under_root option to true.
To set a custom type field in Filebeat using the document_type configuration option.
filebeat:
prospectors:
- paths:
- /var/log/messages
document_type: syslog
This will set the #metadata.type field for use with Logstash whereas a custom field will not.
Trying to configure suricata v2.0.8 with ElasticSearch(v1.5.2)-Logstash(v1.4.2)-Kibana(v4.0.2) on Mac OS X 10.10.3 Yosemite.
suricata.yaml:
# Extensible Event Format (nicknamed EVE) event log in JSON format
- eve-log:
enabled: yes
type: file #file|syslog|unix_dgram|unix_stream
filename: eve.json
# the following are valid when type: syslog above
#identity: "suricata"
#facility: local5
#level: Info ## possible levels: Emergency, Alert, Critical,
## Error, Warning, Notice, Info, Debug
types:
- alert
- http:
extended: yes # enable this for extended logging information
# custom allows additional http fields to be included in eve-log
# the example below adds three additional fields when uncommented
#custom: [Accept-Encoding, Accept-Language, Authorization]
- dns
- tls:
extended: yes # enable this for extended logging information
- files:
force-magic: yes # force logging magic on all logged files
force-md5: yes # force logging of md5 checksums
#- drop
- ssh
#- smtp
#- flow
logstash.conf:
input {
file {
path => ["/var/log/suricata/eve.json"]
sincedb_path => ["/var/lib/logstash/"]
codec => json
type => "SuricataIDPS"
start_position => "beginning"
}
}
filter {
if [type] == "SuricataIDPS" {
date {
match => [ "timestamp", "ISO8601" ]
}
ruby {
code => "if event['event_type'] == 'fileinfo'; event['fileinfo']['type']=event['fileinfo']['magic'].to_s.split(',')[0]; end;"
}
}
if [src_ip] {
geoip {
source => "src_ip"
target => "geoip"
#database => "/usr/local/opt/logstash/libexec/vendor/geoip/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
}
if ![geoip.ip] {
if [dest_ip] {
geoip {
source => "dest_ip"
target => "geoip"
#database => "/usr/local/opt/logstash/libexec/vendor/geoip/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
}
}
}
}
}
output {
elasticsearch {
host => localhost
#protocol => http
}
}
Suricata logs all events successfully into eve.json. When I open kibana in browser, I see no dashboards or any information from suricata... So I assume either logstash doesn't read the data from eve.json or doesn't parse the data to elasticsearch (or both)... Are there any ways to check what's going on?
Turn on a debug output in logstash:
output {
stdout {
codec = rubydebug
}
}
Also, try running your query against Elasticsearch directly (curl) rather than with kibana.
I made an adaptation of the nginx log to the suricata log. I can have the geoip information in the suricata logs. I make the adaptation through swatch and send to a log file configured in filebeat.
Ex:
nginx.access.referrer: ET INFO Session Traversal Utilities for NAT (STUN Binding Request) [**
nginx.access.geoip.location:
{
“lon”: -119.688,
“lat”: 45.8696
}
Use the swatch to read the suricata logs and send them to the shell script that will do the adaptation.
Ex:
echo "$IP - - [$nd4] \"GET $IP2:$PORT2 --- $TYPE HTTP/1.1\" 777 0 \"$CVE\" \"Mozilla/5.0 (NONE) (NONE) NONE\"" >> /var/log/suricata_mod.log
Then configure filebeat.yml:
document_type: nginx-access
paths:
/var/log/suricata_mod.log
Restart filebeat.
Finally configure the logstash:
filter {
if [type] == "nginx-access" {
grok {
match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{HTTPDATE:[nginx][access][time]}\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[$
remove_field => "message"}
mutate {
add_field => { "read_timestamp" => "%{#timestamp}" }}
date {
match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
remove_field => "[nginx][access][time]"}
useragent {
source => "[nginx][access][agent]"
target => "[nginx][access][user_agent]"
remove_field => "[nginx][access][agent]"}
geoip {
source => "[nginx][access][remote_ip]"
target => "[nginx][access][geoip]"
database => "/opt/GeoLite2-City.mmdb"}} } output {
elasticsearch {
hosts => [ "xxx.xxx.xxx.xxx:9200" ]
manage_template => false
document_type => "%{[#metadata][type]}"
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"}}
And restart logstash. In Kibana create a filebeat- * index. Ready.
I followed the steps in this document and I was able to do get some reports on the Shakespeare data.
I want to do the same thing with elastic search remotely installed.I tried configuring the "host" in config file but the queries still run on host as opposed to remote .This is my config file
input {
stdin{
type => "stdin-type" }
file {
type => "accessLog"
path => [ "/Users/akushe/Downloads/requests.log" ]
}
}
filter {
grok {
match => ["message","%{COMMONAPACHELOG} (?:%{INT:responseTime}|-)"]
}
kv {
source => "request"
field_split => "&?"
}
if [lng] {
kv {
add_field => [ "location" , ["%{lng}","%{lat}"]]
}
}else if [lon] {
kv {
add_field => [ "location" , ["%{lon}","%{lat}"]]
}
}
}
output {
elasticsearch {
host => "slc-places-qa-es3001.slc.where.com"
port => 9200
}
}
You need to add protocol => http in to make it use HTTP transport rather than joining the cluster using multicast.