How to write if condition inside of the logstash grok pattern? - elasticsearch

My question is related to logstash grok pattern. I created below pattern that's working fine but the big problem is not string values. Sometimes; "Y" and "age" can be null so my grok pattern not create any log in elasticseach. It is not working properly. I need to tell my grok pattern :
if(age is null || age i empty){
updatefield["age",0]
}
but I don't know how to make it. by the way; I checked many solutions by googling but it is directly related to my problem.
input {
file {
path => ["C:/log/*.log"]
start_position => "beginning"
discover_interval => 10
stat_interval => 10
sincedb_write_interval => 10
close_older => 10
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601}\|"
negate => true
what => "previous"
}
}
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:formattedDate}.* X: %{DATA:X} Y: %{NUMBER:Y} Z: %{DATA:Z} age: %{NUMBER:age:int} "}
}
date {
timezone => "Europe/Istanbul"
match => ["TimeStamp", "ISO8601"]
}
json{
source => "request"
target => "parsedJson"
}
mutate {
remove_field => [ "path","message","tags","#version"]
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => [ "http://localhost:9200" ]
index => "logstash-%{+YYYY.MM}"
}
}

You can check if your fields exists or are empty using conditionals with your filter,
filter {
if ![age] or [age] == "" {
mutate {
update => { "age" => "0" }
}
}
}

Related

Create index based on message field - appname

****Logstash.conf code *******
input {
stdin{
type => "stdin-type"
}
file{
type => "json"
path => [ "C:/prod/*.log", "C:/prod/*/**.log"]
start_position => "beginning"
tags => "prod"
}
file{
type => "json"
path => [ "C:/dev/*.log", "C:/dev/*/**.log"]
start_position => "beginning"
tags => "dev"
}
}
filter {
grok {
match => {
"message" => [ "%{JSON:payload_raw} "]
}
pattern_definitions => {
"JSON" => "{.*$"}
}
json {
source => "payload_raw"
target => "payload"
}
mutate {
remove_field => [ "payload_raw","message" ]
}
date {
match => [ "[payload][datetime]", "yyyy-MM-dd HH:mm:ss,SSS" ]
target => "#timestamp"
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["localhost:9200"]
index => "%{tags}-logs"
}
}
Sample log
{datetime":"2021-08-10 04:11:37,825","servername":"VM-0001","serverip":"(null)","process":"2404","thread":"4","level":"DEBUG","appname":"Dev-Email","page":"Program.cs"}
Given the sample document your shared, your elasticsearch output needs to look like this:
elasticsearch {
hosts => ["localhost:9200"]
index => "%{appname}-logs"
}
Also know that index names are not allowed to contain uppercase letters, so Dev-Email will need to be lowercased (using the mutate/lowercase filter) before being used as the index name.

logstash don't report all the events

i could see some events are missing while reporting logs to elastic search. Take an example i am sending 5 logs event only 4 or 3 are reporting.
Basically i am using logstash 7.4 to read my log messages and store the information on elastic search 7.4. below is my logstash configuration
input {
file {
type => "web"
path => ["/Users/a0053/Downloads/logs/**/*-web.log"]
start_position => "beginning"
sincedb_path => "/tmp/sincedb_file"
codec => multiline {
pattern => "^(%{MONTHDAY}-%{MONTHNUM}-%{YEAR} %{TIME}) "
negate => true
what => previous
}
}
}
filter {
if [type] == "web" {
grok {
match => [ "message","(?<frontendDateTime>%{MONTHDAY}-%{MONTHNUM}-%{YEAR} %{TIME})%{SPACE}(\[%{DATA:thread}\])?( )?%{LOGLEVEL:level}%{SPACE}%{USERNAME:zhost}%{SPACE}%{JAVAFILE:javaClass} %{USERNAME:orgId} (?<loginId>[\w.+=:-]+#[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:[.](?:[0-9A-Za-z][0-9A-Za-zā€Œā€‹-]{0,62}))*) %{GREEDYDATA:jsonstring}"]
}
json {
source => "jsonstring"
target => "parsedJson"
remove_field=>["jsonstring"]
}
mutate {
add_field => {
"actionType" => "%{[parsedJson][actionType]}"
"errorMessage" => "%{[parsedJson][errorMessage]}"
"actionName" => "%{[parsedJson][actionName]}"
"Payload" => "%{[parsedJson][Payload]}"
"pageInfo" => "%{[parsedJson][pageInfo]}"
"browserInfo" => "%{[parsedJson][browserInfo]}"
"dateTime" => "%{[parsedJson][dateTime]}"
}
}
}
}
output{
if "_grokparsefailure" in [tags]
{
elasticsearch
{
hosts => "localhost:9200"
index => "grokparsefailure-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch
{
hosts => "localhost:9200"
index => "zindex"
}
}
stdout{codec => rubydebug}
}
As keep on new logs are writing to log files, i could see a difference of log counts.
Any suggestions would be appreciated.

Match multiple patterns in Logstash?

I have two types of error messages in the below format:
[2017-05-25 01:00:00,647][ERROR][marvel.agent.exporter.local] local exporter [default_local] - failed to delete indices
RemoteTransportException[[data-0][10.0.0.8:9300][indices:admin/delete]]; nested: IndexNotFoundException[no such index];
[2017-05-18 00:00:06,339][DEBUG][action.admin.indices.create] [data-2] [data-may-2017,data-apr-2017,data-mar-2017] failed to create
[data-may-2017,data-apr-2017,data-mar-2017]
My logstash configuration is like this:
input {
file {
path => "D:\logstash\logstash-2.4.0\bin\logs.txt"
start_position => "beginning"
codec => multiline {
pattern => "^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]"
negate => true
what => "previous"
}
}
}
filter {
grok {
match => [ "message", "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}\]%{SPACE}\[%{DATA:SERVERNAME}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)", "message", "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}%{SPACE}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)"]
}
}
output {
stdout { codec => rubydebug }
}
For Both the logs it is taking only the first grok pattern. Why it is not taking the second one?
Seems my first grok pattern is matching all the logs , so thats why logstash is taking only the first pattern. So that i had used the below config with if condition which is working fine.
input {
file {
path => "D:\logstash\logstash-2.4.0\bin\logs.txt"
start_position => "beginning"
type => "log"
codec => multiline {
pattern => "^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]"
negate => true
what => "previous"
}
}
}
filter {
if [type] == "log" {
grok {
match => [ "message", "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}%{SPACE}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)"]
}
# DEBUG Logs
if "grokked" not in [tags] and "DEBUG" == [LEVEL] {
grok { match => [ "ERRORMESSAGE", "(?m)^\[%{DATA:SERVERNAME}\]" ]
add_tag => [ "Debug Logs", "grokked" ]
tag_on_failure => [ ]
}
}
}
}
output {
stdout { codec => rubydebug }
}
Your question was:
Why it is not taking the second one?
Answer is here:
filter {
grok {
match => [ "message", "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}\]%{SPACE}\[%{DATA:SERVERNAME}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)", "message", "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}%{SPACE}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)"]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------^
}
}
You don't have to specify source multiple times, just once.
What you did now was:
["message", "pattern", "message", "pattern"]
While in reality it has to be:
["message", "pattern", "pattern", ..., "pattern"]

:reason=>"Something is wrong with your configuration." GeoIP.dat Mutate Logstash

I have the following configuration for logstash.
There are 3 parts to this one is a generallog which we use for all applications they land in here.
second part is the application stats where in which we have a specific logger which will be configured to push the application statistics
third we have is the click stats when ever an event occurs on client side we may want to push it to the logstash on the upd address.
all 3 are udp based, we also use log4net to to send the logs to the logstash.
the base install did not have a GeoIP.dat file so got the file downloaded from the https://dev.maxmind.com/geoip/legacy/geolite/
have put the file in the /opt/logstash/GeoIPDataFile with a 777 permissions on the file and folder.
second thing is i have a country name and i need a way to show how many users form each country are viewing the application in last 24 hours.
so for that reason we also capture the country name as its in their profile in the application.
now i need a way to get the geo co-ordinates to use the tilemap in kibana.
What am i doing wrong.
if i take the geoIP { source -=> "country" section the logstash works fine.
when i check the
/opt/logstash/bin/logstash -t -f /etc/logstash/conf.d/logstash.conf
The configuration file is ok is what i receive. where am i going worng?
Any help would be great.
input {
udp {
port => 5001
type => generallog
}
udp {
port => 5003
type => applicationstats
}
udp {
port => 5002
type => clickstats
}
}
filter {
if [type] == "generallog" {
grok {
remove_field => message
match => { message => "(?m)%{TIMESTAMP_ISO8601:sourcetimestamp} \[%{NUMBER:threadid}\] %{LOGLEVEL:loglevel} +- %{IPORHOST:requesthost} - %{WORD:applicationname} - %{WORD:envname} - %{GREEDYDATA:logmessage}" }
}
if !("_grokparsefailure" in [tags]) {
mutate {
replace => [ "message" , "%{logmessage}" ]
replace => [ "host" , "%{requesthost}" ]
add_tag => "generalLog"
}
}
}
if [type] == "applicationstats" {
grok {
remove_field => message
match => { message => "(?m)%{TIMESTAMP_ISO8601:sourceTimestamp} \[%{NUMBER:threadid}\] %{LOGLEVEL:loglevel} - %{WORD:envName}\|%{IPORHOST:actualHostMachine}\|%{WORD:applicationName}\|%{NUMBER:empId}\|%{WORD:regionCode}\|%{DATA:country}\|%{DATA:applicationName}\|%{NUMBER:staffapplicationId}\|%{WORD:applicationEvent}" }
}
geoip {
source => "country"
target => "geoip"
database => "/opt/logstash/GeoIPDataFile/GeoIP.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
if !("_grokparsefailure" in [tags]) {
mutate {
add_tag => "applicationstats"
add_tag => [ "eventFor_%{applicationName}" ]
}
}
}
if [type] == "clickstats" {
grok {
remove_field => message
match => { message => "(?m)%{TIMESTAMP_ISO8601:sourceTimestamp} \[%{NUMBER:threadid}\] %{LOGLEVEL:loglevel} - %{IPORHOST:remoteIP}\|%{IPORHOST:fqdnHost}\|%{IPORHOST:actualHostMachine}\|%{WORD:applicationName}\|%{WORD:envName}\|(%{NUMBER:clickId})?\|(%{DATA:clickName})?\|%{DATA:clickEvent}\|%{WORD:domainName}\\%{WORD:userName}" }
}
if !("_grokparsefailure" in [tags]) {
mutate {
add_tag => "clicksStats"
add_tag => [ "eventFor_%{clickName}" ]
}
}
}
}
output {
if [type] == "applicationstats" {
elasticsearch {
hosts => "localhost:9200"
index => "applicationstats-%{+YYYY-MM-dd}"
template => "/opt/logstash/templates/udp-applicationstats.json"
template_name => "applicationstats"
template_overwrite => true
}
}
else if [type] == "clickstats" {
elasticsearch {
hosts => "localhost:9200"
index => "clickstats-%{+YYYY-MM-dd}"
template => "/opt/logstash/templates/udp-clickstats.json"
template_name => "clickstats"
template_overwrite => true
}
}
else if [type] == "generallog" {
elasticsearch {
hosts => "localhost:9200"
index => "generallog-%{+YYYY-MM-dd}"
template => "/opt/logstash/templates/udp-generallog.json"
template_name => "generallog"
template_overwrite => true
}
}
else{
elasticsearch {
hosts => "localhost:9200"
index => "logstash-%{+YYYY-MM-dd}"
}
}
}
As per the error message, the mutation which you're trying to do could be wrong. Could you please change your mutate as below:
mutate {
convert => { "geoip" => "float" }
convert => { "coordinates" => "float" }
}
I guess you've given the mutate as an array, and it's a hash type by origin. Try converting both the values individually. Your database path for geoip seems to be fine in your filter. Is that the whole error which you've mentioned in the question? If not update the question with the whole error if possible.
Refer here, for in depth explanations.

logstash: multiple logfiles with different pattern

We want to set up a server for logstash for a couple of different project in our company. Now I try to enable them in Kibana. My question is:
If I have different patterns of the logfiles, how can I build for them a filter?
example: logstash.conf:
input {
file {
type => "A"
path => "/home/logstash/A/*"
start_position => "beginning"
}
file {
type => "B"
path => "/home/logstash/B*"
start_position => "beginning"
}
}
filter {
multiline {
pattern => "^%{TIMESTAMP_ISO8601}"
negate => true
what => "previous"
}
grok {
type => A
match => [ "message", "%{TIMESTAMP_ISO8601:logdate} %{DATA:thread %{LOGLEVEL:level}\s*%{DATA:logger_name}\s*-\s*%{GREEDYDATA:log_text}"]
add_tag => [ "level_%{level}" ]
}
date {
match => ["logdate", "YYYY-MM-dd HH:mm:ss,SSS"]
}
grok {
type => B
match => [ any other pattern ...
}
}
output {
elasticsearch { embedded => true }
}
do I have to create for each project (A,B,C,...) an own filter, and what do I have to do, when I have for each project again different pattern of the logfiles?
You only need to create a filter for all projects.
For Logstash 1.3.3, You can use if statement to distinct each project grok. For example,
filter {
multiline {
pattern => "^%{TIMESTAMP_ISO8601}"
negate => true
what => "previous"
}
if [type] == "A" {
grok {
match => [ any other pattern ...
}
}
else if [type] == "B" {
grok {
match => [ any other pattern ...
}
}
}
Hope this can help you.

Resources