Logstash version 6.5.4
I want to create jobExecutionTime field when status is COMPLETE and set its value as current_timestamp-created_timestamp.
These are few lines from my config file.
match => { "message" => '%{DATA:current_timestamp},%{WORD:status},%{DATA:created_timestamp}}
if [status] == "COMPLETE" {
mutate {
add_field => [ "jobExecutionTime" , "null" ]
}
ruby {
code => "event.set('jobExecutionTime', event.get('current_timestamp') - event.get('created_timestamp'))"
}
}
This my input
"created_timestamp" => "2022-07-10 23:50:03.644"
"current_timestamp" => "2022-07-10 23:50:03.744"
"status" => "COMPLETE"
I am getting this as output
"jobExecutionTime" => "null",
"exportFrequency" => "RECURRENT",
"successfulImportMilestone" => 0,
"tags" => [
[0] "_rubyexception"
],
Here jobExecutionTime is set to null rather than concerned value
Your [created_timestamp] and [current_timestamp] fields are strings. You cannot do math on a string, you need to convert it an object type that you can do math on. In this case you should use date filters to convert them to LogStash::Timestamp objects
If you add
date { match => [ "created_timestamp", "ISO8601" ] target => "created_timestamp" }
date { match => [ "current_timestamp", "ISO8601" ] target => "current_timestamp" }
to your filter section then your ruby filter will work as-is, and you will get
"created_timestamp" => 2022-07-11T03:50:03.644Z,
"current_timestamp" => 2022-07-11T03:50:03.744Z,
"jobExecutionTime" => 0.1
Related
In the logstash config file I am trying to just get the XML data to parse.
This is my config file:
input {
file {
path => "/home/elastic-stack/logstash-7.3.2/event-data/telmetry.log"
start_position => "beginning"
type => "sandbox-out"
codec => multiline {
pattern => "^</datastore-contents-xml>"
negate => "true"
what => "next"
}
}
http {
host => "127.0.0.1"
port => 8080
type => "sandbox-out"
}
}
filter {
grok {
match => { "message" => "\[%{USER:host_name} %{IP:ip_address} %{USER:session-id} %{NUMBER:session-id-num}\]"}
}
grok {
match => { "message" => "\Subscription Id \: %{BASE16NUM:subcription-id:int}"}
}
grok {
match => { "message" => "\Event time \: %{TIMESTAMP_ISO8601:event-time}"}
}
grok {
match => {"message" => "\<%{USERNAME:Statistic}\>"}
}
mutate {
remove_field => ["headers", "host_name", "session-id","message"]
}
date {
match => ["timestamp","dd/MMM/yyyy:HH:mm:ss Z"]
}
ruby { code => 'event.set("justXml", event.get("message").match(/.+(<datastore-contents-xml.*)/m)[1])' }
xml {
#remove_namespaces => "true"
#not even the namspace option is working to access the http link
source => "justXml"
target => "xml-content"
#force_array => "false"
xpath => [
"//*[name()='datastore-contents-xml']/*[name()='memory-statistics']/*[name()='memory-statistic'][1]/*[name()='name']/text()" , "name" ,
"//*[name()='datastore-contents-xml']/*[name()='memory-statistics']/*[name()='memory-statistic'][1]/*[name()='total-memory']/text()" , "total-memory",
"//*[name()='datastore-contents-xml']/*[name()='memory-statistics']/*[name()='memory-statistic'][1]/*[name()='used-memory']/text()" , "used-memory",
"//*[name()='datastore-contents-xml']/*[name()='memory-statistics']/*[name()='memory-statistic'][1]/*[name()='free-memory']/text()" , "free-memory" ,
"//*[name()='datastore-contents-xml']/*[name()='memory-statistics']/*[name()='memory-statistic'][1]/*[name()='lowest-memory']/text()" , "lowest-memory" ,
"//*[name()='datastore-contents-xml']/*[name()='memory-statistics']/*[name()='memory-statistic'][1]/*[name()='highest-memory']/text()" , "highest-memory"
]
#logstash is not dectecting any of these xpaths in the config
}
mutate {
convert => {
"total-memory" => "integer"
"used-memory" => "integer"
"free-memory" => "integer"
"lowest-memory" => "integer"
"highest-memory" => "integer"
}
}
}
output {
stdout {
codec => rubydebug
}
file {
path => "%{type}_%{+dd_MM_yyyy}.log"
}
}
Desired output:
{
"ip_address" => "10.10.20.30",
"subcription-id" => 2147483650,
"event-time" => "2019-09-12 13:13:30.290000+00:00",
"host" => "127.0.0.1",
"Statistic" => "memory-statistic",
"type" => "sandbox-out",
"#version" => "1",
"#timestamp" => 2019-09-26T10:03:00.620Z,
"session-id-num" => "35"
"yang-model" => "http://cisco.com/ns/yang/Cisco-IOS-XE-memory-oper"
"name" => "Processor"
"total-memory" => 2238677360
"used-memory" => 340449924
"free-memory" => 1898227436
"lowest-usage" => 1897220640
"highest-usage" => 1264110388
}
ERROR:
[2019-09-27T09:18:55,622][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method `match' for nil:NilClass
/home/elastic-stack/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"ip_address" => "10.10.20.30",
"subcription-id" => 2147483650,
"session-id-num" => "35",
"tags" => [
[0] "_rubyexception"
],
"Statistic" => "memory-statistic",
"event-time" => "2019-09-12 13:13:30.290000+00:00",
"type" => "sandbox-out",
"#version" => "1",
"host" => "127.0.0.1",
"#timestamp" => 2019-09-27T07:18:54.868Z
By the error I can already know that the problem is with the ruby filter but I do not know how to resolve it.
This data generate by Cisco Telemetry and I am trying to ingest it using Elastic Stack.
The error seems to be that the event has no message field, so you cannot call match on a non existing thing.
I see you are calling match on the message field in this ruby code:
ruby { code => 'event.set("justXml", event.get("message").match(/.+(<datastore-contents-xml.*)/m)[1])' }
However you are removing the message field from the event a few lines earlier:
mutate {
remove_field => ["headers", "host_name", "session-id","message"]
}
Solution is to remove the message field only when you don't need it anymore, I would move the remove_field mutate to the end of the filter block.
One more suggestion if I may add. You have multiple grok filters running on the same, message field:
grok {
match => { "message" => "\[%{USER:host_name} %{IP:ip_address} %{USER:session-id} %{NUMBER:session-id-num}\]"}
}
grok {
match => { "message" => "\Subscription Id \: %{BASE16NUM:subcription-id:int}"}
}
grok {
match => { "message" => "\Event time \: %{TIMESTAMP_ISO8601:event-time}"}
}
grok {
match => {"message" => "\<%{USERNAME:Statistic}\>"}
}
This can be simplified into this (you can check to the Grok filter docs:
grok {
break_on_match => false,
match => {
"message" => [
"\[%{USER:host_name} %{IP:ip_address} %{USER:session-id} %{NUMBER:session-id-num}\]",
"\Subscription Id \: %{BASE16NUM:subcription-id:int}",
"\Event time \: %{TIMESTAMP_ISO8601:event-time}",
"\<%{USERNAME:Statistic}\>"
]
}
}
This way you need only one instance of the grok filter, as it will go through the patterns in the list and because of break_on_match=>false it will not finish after the first successful match, but will make sure to extract all fields it can based on all the patterns in the list.
My question is related to logstash grok pattern. I created below pattern that's working fine but the big problem is not string values. Sometimes; "Y" and "age" can be null so my grok pattern not create any log in elasticseach. It is not working properly. I need to tell my grok pattern :
if(age is null || age i empty){
updatefield["age",0]
}
but I don't know how to make it. by the way; I checked many solutions by googling but it is directly related to my problem.
input {
file {
path => ["C:/log/*.log"]
start_position => "beginning"
discover_interval => 10
stat_interval => 10
sincedb_write_interval => 10
close_older => 10
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601}\|"
negate => true
what => "previous"
}
}
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:formattedDate}.* X: %{DATA:X} Y: %{NUMBER:Y} Z: %{DATA:Z} age: %{NUMBER:age:int} "}
}
date {
timezone => "Europe/Istanbul"
match => ["TimeStamp", "ISO8601"]
}
json{
source => "request"
target => "parsedJson"
}
mutate {
remove_field => [ "path","message","tags","#version"]
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => [ "http://localhost:9200" ]
index => "logstash-%{+YYYY.MM}"
}
}
You can check if your fields exists or are empty using conditionals with your filter,
filter {
if ![age] or [age] == "" {
mutate {
update => { "age" => "0" }
}
}
}
I am trying to add a field to set #timestamp timezone match my timestamp
I have tried adding timezone to filter, but it doesn't help.
original timestamp looks like ,
"#timestamp" => "2016-06-08T01:36:42.655Z",
My date filter is:
mutate {
add_field => [ "localtime", "%{#timestamp}" ]
}
date {
locale => "en_US"
timezone => "America/New_York"
match => [ "localtime", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"]
target => "localtime"
}
and get the warning _dateparsefailure
am i missing anything ?
can anyone please help me out? Thanks.
Your match needs to work on a different field than #timestamp, which is not a string and is thus not parseable. Since you've copied #timestamp into datetime, you can now use datetime in your match.
mutate {
add_field => [ "datetime", "%{#timestamp}" ]
}
date {
locale => "en_US"
timezone => "America/New_York"
match => [ "datetime", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"]
#match => [ "datetime", "ISO8601"] # <- you can also use this
target => "datetime"
}
In my log files, I have data that represents a the hierarchy of items, much like an http log file might show the hierarchy of a website.
I may have data such as this
41 2016-01-01 01:41:32-500 show:category:all
41 2016-01-01 04:11:20-500 show:category:animals
42 2016-01-02 01:41:32-500 show:item:wallaby
42 2016-01-02 01:41:32-500 show:home
and I would have 3 items in here... %{NUMBER:terminal} %{TIMESTAMP_ISO8601:ts} and (?<info>([^\r])*)
I parse the info data into an array using mutate and split to convert lvl1:lvl2:lvl3 into ['lvl1','lvl2','lvl3'].
I'm interested in aggregating the data to get counts at various levels easily, such as counting all records where info[0] is the same or where info[0] and info[1] are the same. (and be able to select time range and terminal)
Is there a way to set up kibana to visualize this kind of information?
Or should I change the way the filter is matching the data to make the data easier to access?
the depth of levels varies but I can be pretty certain that the max levels are 5, so I could parse the text into various fields lvl1 lvl2 lvl3 lvl4 lvl5 instead of putting them in an array.
As per your question, I agree with your way of parsing data. But I would like to add on more to make it directly aggregatable & visualize using Kibana.
The approach should be :-
Filter the data using %{NUMBER:terminal} %{TIMESTAMP_ISO8601:ts} and (?([^\r])*) {As per information given by you}
Mutate
Filter
Then after using mutate & filter you will get data in terms of array {as you have mentioned}
Now you can add a field as level 1 by mentioning add_field => [ "fieldname", "%{[arrayname][0]}" ]
Now you can add a field as level 2 by mentioning add_field => [ "fieldname", "%{[arrayname][1]}" ]
Now you can add a field as level 3 by mentioning add_field => [ "fieldname", "%{[arrayname][2]}" ]
Then you can directly use Kibana to visualize such information.
my solution
input {
file {
path => "C:/Temp/zipped/*.txt"
start_position => beginning
ignore_older => 0
sincedb_path => "C:/temp/logstash_temp2.sincedb"
}
}
filter {
grok {
match => ["message","^%{NOTSPACE}\[%{NUMBER:terminal_id}\] %{NUMBER:log_level} %{NUMBER} %{TIMESTAMP_ISO8601:ts} \[(?<facility>([^\]]*))\] (?<lvl>([^$|\r])*)"]
}
mutate {
split => ["lvl", ":"]
add_field => {"lvl_1" => "%{lvl[0]}"}
add_field => {"lvl_2" => "%{lvl[1]}"}
add_field => {"lvl_3" => "%{lvl[2]}"}
add_field => {"lvl_4" => "%{lvl[3]}"}
add_field => {"lvl_5" => "%{lvl[4]}"}
add_field => {"lvl_6" => "%{lvl[5]}"}
add_field => {"lvl_7" => "%{lvl[6]}"}
add_field => {"lvl_8" => "%{lvl[7]}"}
lowercase => [ "terminal_id" ] # set to lowercase so that it can be used for index - additional filtering may be required
}
date {
match => ["ts", "YYYY-MM-DD HH:mm:ssZZ"]
}
}
filter {
if [lvl_1] =~ /%\{lvl\[0\]\}/ {mutate {remove_field => [ "lvl_1" ]}}
if [lvl_2] =~ /%\{lvl\[1\]\}/ {mutate {remove_field => [ "lvl_2" ]}}
if [lvl_3] =~ /%\{lvl\[2\]\}/ {mutate {remove_field => [ "lvl_3" ]}}
if [lvl_4] =~ /%\{lvl\[3\]\}/ {mutate {remove_field => [ "lvl_4" ]}}
if [lvl_5] =~ /%\{lvl\[4\]\}/ {mutate {remove_field => [ "lvl_5" ]}}
if [lvl_6] =~ /%\{lvl\[5\]\}/ {mutate {remove_field => [ "lvl_6" ]}}
if [lvl_7] =~ /%\{lvl\[6\]\}/ {mutate {remove_field => [ "lvl_7" ]}}
if [lvl_8] =~ /%\{lvl\[7\]\}/ {mutate {remove_field => [ "lvl_8" ]}}
mutate{
remove_field => [ "lvl","host","ts" ] # do not keep this data
}
}
output {
if [facility] == "mydata" {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-mydata-%{terminal_id}-%{+YYYY.MM.DD}"
}
} else {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-other-%{terminal_id}-%{+YYYY.MM.DD}"
}
}
# stdout { codec => rubydebug }
}
I am using genjdbc input plugin for Logstash to get data from a DB2 database. It works perfectly, I get in Kibana all the database columns as fields.
The problem I have is that in Kibana all fields are string type, and I want the numeric fields to be integers. I have tried the following code, but the result is the same that if no filter clause exists.
Can someone help me solving this? Thanks in advance!
The logstash.conf code:
input {
genjdbc {
jdbcHost => "XXX.XXX.XXX.XXX"
jdbcPort => "51260"
jdbcTargetDB => "db2"
jdbcDBName => "XXX"
jdbcUser => "XXX"
jdbcPassword => "XXX"
jdbcDriverPath => "C:\...\db2jcc4.jar"
jdbcSQLQuery => "SELECT * FROM XXX1"
jdbcTimeField => "LOGSTAMP"
jdbcPStoreFile => "C:\elk\logstash\bin\db2.pstore"
jdbcURL => "jdbc:db2://XXX.XXX.XXX.XXX:51260/XXX"
type => "table1"
}
genjdbc {
jdbcHost => "XXX.XXX.XXX.XXX"
jdbcPort => "51260"
jdbcTargetDB => "db2"
jdbcDBName => "XXX"
jdbcUser => "XXX"
jdbcPassword => "XXX"
jdbcDriverPath => "C:\...\db2jcc4.jar"
jdbcSQLQuery => "SELECT * FROM XXX2"
jdbcTimeField => "LOGSTAMP"
jdbcPStoreFile => "C:\elk\logstash\bin\db2.pstore"
jdbcURL => "jdbc:db2://XXX.XXX.XXX.XXX:51260/XXX"
type => "table2"
}
}
filter {
mutate {
convert => [ "T1", "integer" ]
convert => [ "T2", "integer" ]
convert => [ "T3", "integer" ]
}
}
output {
if [type] == "table1" {
elasticsearch {
host => "localhost"
protocol => "http"
index => "db2_1-%{+YYYY.MM.dd}"
}
}
if [type] == "table2" {
elasticsearch {
host => "localhost"
protocol => "http"
index => "db2_2-%{+YYYY.MM.dd}"
}
}
}
What you have should work as long as the fields you are trying to convert to integer are names T1,T2,T3 and you are inserting into an index that doesn't have any data. If you already have data in the index, you'll need to delete the index so that logstash can recreate it with the correct mapping.