I'm New in ELK. I have created index with name "ordersatus" which store the status published by logistic partner.
whenever logistic partner update the order status then new status is pushed into elasticseach.
Now every order is having multiple enteries with order status like "ORDER CONFIRM" , "APPOINTMENT SCHEDULED" , "OUT FOR DELIVERY" etc..
Problem arise when i need to see how many order are in which status.
Total Order Count is 2, but in order status i get total count 4. because it count older values too. as you can see in attached screenshot.
I Want to Display all unique order status along with the order count having that status.
i.e
ORDER STATUS | TOTAL COUNT
APPOINTMENT_CONFIRMED | 1
ASSIGN_FOR_DELIVERY | 1
as of now its Displaying order status "CONFIRMED" with count 2. which is older value of these 2 orders.
Screenshot1
Screenshot2
You want the distinct statuses and their count in the table of the second screenshot? You have two selections in the buckets — timestamp and status: Remove the timestamp, since this will split into every unique timestamp. What you want is only the status.
This is a different dataset, but you get the idea:
As a workaround, you can add the below to the logstash config for adding weights for each status. When the status transforming from the previous status, add -1 to the previous status weight and add 1 to the new status weight.
if [status] == "ORDER CONFIRM" {
mutate { "add_field" => { "order_confirm_weight" => "1" } }
mutate { convert => { "order_confirm_weight" => "integer" }}
} else if [status] == "APPOINTMENT SCHEDULED" {
mutate { "add_field" => { "order_confirm_weight" => "-1" } }
mutate { convert => { "order_confirm_weight" => "integer" }}
mutate { "add_field" => { "appointment_scheduled_weight" => "1" } }
mutate { convert => { "appointment_scheduled_weight" => "integer" }}
} else if [status] == "OUT FOR DELIVERY" {
mutate { "add_field" => { "appointment_scheduled_weight" => "-1" } }
mutate { convert => { "appointment_scheduled_weight" => "integer" }}
mutate { "add_field" => { "out_for_delivery_weight" => "1" } }
mutate { convert => { "out_for_delivery_weight" => "integer" }}
}
Once the configuration is added to the logstash, you can get the count of each status in kibana visualization using the below aggregating functions.
ORDER CONFIRM Count = Sum (order_confirm_weight)
APPOINTMENT SCHEDULED Count = Sum (appointment_scheduled_weight)
OUT FOR DELIVERY Count = Sum (out_for_delivery_weight)
Related
4 fields (warnTags、warnSlrs、warnActions、denyMsg) fields need to be separated by semicolon(;)
Raw String
{ "waf": {
"warnTags": "OWASP_CRS/WEB_ATTACK/SQL_INJECTION;OWASP_CRS/WEB_ATTACK/XSS;OWASP_CRS/WEB_ATTACK/XSS;OWASP_CRS/WEB_ATTACK/XSS;OWASP_CRS/WEB_ATTACK/SPECIAL_CHARS;OWASP_CRS/WEB_ATTACK/SQL_INJECTION",
"policy": "bot_77598",
"warnSlrs": "ARGS:wvstest;ARGS:wvstest;ARGS:wvstest;ARGS:wvstest;ARGS:wvstest;ARGS:wvstest",
"riskTuples": ":-973305-973333-973335",
"warnActions": "2;2;2;2;2;2",
"denyActions": "3",
"warnMsg": "SQL Injection Attack;XSS Attack Detected;IE XSS Filters - Attack Detected;IE XSS Filters - Attack Detected;Restricted SQL Character Anomaly Detection Alert - Total # of special characters exceeded;Classic SQL Injection Probes 1/2",
"riskGroups": ":XSS-ANOMALY",
"warnRules": "950901;973305;973333;973335;981173;981242",
"denyMsg": "Anomaly Score Exceeded for Cross-Site Scripting",
"ver": "2.0",
"denyData": "VmVjdG9yIFNjb3JlOiBx",
"riskScores": ":-5-5-2",
"warnData": "eHNzdGFnPigpbG9jeHNz;amF2YXNYcm"
} }
Expected Output Result
{
"waf": {
"warnTags": "OWASP_CRS/WEB_ATTACK/SQL_INJECTION",
"policy": "bot_77598",
"warnSlrs": "ARGS:wvstest",
"riskTuples": ":-973305-973333-973335",
"warnActions": "2",
"denyActions": "3",
"warnMsg": "SQL Injection Attack",
"riskGroups": ":XSS-ANOMALY",
"warnRules": "950901",
"denyMsg": "Anomaly Score Exceeded for Cross-Site Scripting",
"ver": "2.0",
"denyData": "VmVjdG9yIFNjb3JlOiBx",
"riskScores": ":-5-5-2",
"warnData": "eHNzdGFnPigpbG9jeHNz;amF2YXNYcm"
}
}
{
"waf": {
"warnTags": "OWASP_CRS/WEB_ATTACK/XSS",
"policy": "bot_77598",
"warnSlrs": "ARGS:wvstest",
"riskTuples": ":-973305-973333-973335",
"warnActions": "2",
"denyActions": "3",
"warnMsg": "XSS Attack Detected",
"riskGroups": ":XSS-ANOMALY",
"warnRules": "973305",
"denyMsg": "Anomaly Score Exceeded for Cross-Site Scripting",
"ver": "2.0",
"denyData": "VmVjdG9yIFNjb3JlOiBx",
"riskScores": ":-5-5-2",
"warnData": "eHNzdGFnPigpbG9jeHNz;amF2YXNYcm"
}
}
filter {
ruby {
code => "
#info = []
events = event.to_hash
#warnTags = events['waf']['warnTags'].split(';')
#warnMsgs = events['waf']['warnMsg'].split(';')
#warnActions = events['waf']['warnActions'].split(';')
#warnRules = events['waf']['warnRules'].split(';')
#list = #warnTags.zip( #warnMsgs, #warnActions, #warnRules )
#list.each do |tag, msg, action, rule|
detail = {
'tag' => tag,
'msg' => msg,
'action' => action,
'rule' => rule
}
#info.push(detail)
end
event.remove('[waf][warnTags]')
event.remove('[waf][warnMsg]')
event.remove('[waf][warnActions]')
event.remove('[waf][warnRules]')
event.set('[waf][info]', #info)
"
}
split {
field => "[waf][info]"
}}
The config below should be along the lines of what you need. It includes parsing as json at the outset which you may not need depending on prior steps in your pipeline. Essentially this will split the warnTags field on ; to begin with; that will result in warnTags being an array nested within one object. The output of the string split is passed in the to higher level split filter which will create multiple output events splitting on input field, in this case warnTags (again). Hope this helps!
[EDIT: Added warnSlrs as second split field]
filter {
json {
source => "message"
}
mutate {
split => {"[waf][warnTags]" => ";"}
}
mutate {
split => {"[waf][warnSlrs]" => ";"}
}
split {
field => "[waf][warnTags]"
}
split {
field => "[waf][warnSlrs]"
}
}
I have two questions;
parsing xml data & adding it to an array in a record in an index
checking for an existing record in an index and if it exists add the new data of that record to the array of the existing record
I have an jdbc input that has an xml column,
input {
jdbc {
....
statement => "SELECT event_xml....
}
}
then an xml filter to parse the data,
How do i make the the last 3 xpaths to be an array? Do i need a mutate or ruby filter? I cant seem to figure it out
filter {
xml {
source => "event_xml"
remove_namespaces => true
store_xml => false
force_array => false
xpath => [ "/CaseNumber/text()", "case_number" ]
xpath => [ "/FormName/text()", "[conversations][form_name]" ]
xpath => [ "/EventDate/text()", "[conversations][event_date]" ]
xpath => [ "/CaseNote/text()", "[conversations][case_note]" ]
}
}
so it would something like this look like this in the Elastic search.
{
"case_number" : "12345",
"conversations" :
[
{
"form_name" : "form1",
"event_date" : "2019-01-09T00:00:00Z",
"case_note" : "this is a case note"
}
]
}
So second question is, if there is already a unique case_number of "12345" instead of creating a new record for this add the new xml values to the conversations array. so it would look like this
{
"case_number" : "12345",
"conversations" : [
{
"form_name" : "form1",
"event_date" : "2019-01-09T00:00:00Z",
"case_note" : "this is a case note"
},
{
"form_name" : "form2",
"event_date" : "2019-05-09T00:00:00Z",
"case_note" : "this is another case note"
}
]
}
my output filter
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "cases"
manage_template => false
}
}
Is this possible? thanks
this ruby filter created the array
ruby {
code => '
event.set("conversations", [Hash[
"publish_event_id", event.get("publish_event_id"),
"form_name", event.get("form_name"),
"event_date", event.get("event_date"),
"case_note", event.get("case_note")
]])
'
}
for the output was resolved by
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "cases"
document_id => "%{case_number}"
action => "update"
doc_as_upsert => true
script => "
boolean recordExists = false;
for (int i = 0; i < ctx._source.conversations.length; i++)
{
if(ctx._source.conversations[i].publish_event_id == params.event.get('conversations')[0].publish_event_id)
{
recordExists = true;
}
}
if(!recordExists){
ctx._source.conversations.add(params.event.get('conversations')[0]);
}
"
manage_template => false
}
}
I'm receving events in Logstash containing measurement, values and tags. I do not know ahead of time what field there are and what tags. So i wanted to do something like this:
input {
http {}
}
filter {
ruby {
code => '
tags = event.get("stats_tags").split(",")
samples = event.get("stats_samples").split(" ")
datapoints = {}
samples.each {|s|
splat = s.split(" ")
datapoints[splat[0]] = splat[1]
}
event.set("[#metadata][stats-send-as-tags]", tags)
event.set("[#metadata][stats-datapoints]", datapoints)
'
}
}
output {
influxdb {
host => "influxdb"
db => "events_db"
measurement => measurement
send_as_tags => [#metadata][stats-send-as-tags]
data_points => [#metadata][stats-datapoints]
}
}
But this produce error. After much googling to no avail i'm starting to think this is imposible.
Is there a way to pass hash and array from event field to output/filter configuration?
EDIT: If i doublequote it, the error i'm getting is
output {
influxdb {
# This setting must be a hash
# This field must contain an even number of items, got 1
data_points => "[#metadata][stats-datapoints]"
...
}
}
I have a log with a format similar to:
name=johnny amount=30 uuid=2039248934
The problem is I am using this parser on multiple log files with each basically containing numerous kv pairs.
Is there a way to recognize when values are integers and cast them as such without having to use mutate on every single key value pair?(Rather than a string)
I found this link but it was very vague in where the template json file was suppose to go and how I was to go about using it.
Can kv be told to auto-detect numeric values and emit them as numeric JSON values?
You can use ruby plugin to do it.
input {
stdin {}
}
filter {
ruby {
code => "
fieldArray = event['message'].split(' ');
for field in fieldArray
name = field.split('=')[0];
value = field.split('=')[1];
if value =~ /\A\d+\Z/
event[name] = value.to_i
else
event[name] = value
end
end
"
}
}
output {
stdout { codec => rubydebug }
}
First, split the message to an array by SPACE.
Then, for each k,v mapping, check whether the value is numberic, if YES, convert it to Integer.
Here is the sample output for your input:
{
"message" => "name=johnny amount=30 uuid=2039248934",
"#version" => "1",
"#timestamp" => "2015-06-25T08:24:39.755Z",
"host" => "BEN_LIM",
"name" => "johnny",
"amount" => 30,
"uuid" => 2039248934
}
Update Solution for Logstash 5:
input {
stdin {}
}
filter {
ruby {
code => "
fieldArray = event['message'].split(' ');
for field in fieldArray
name = field.split('=')[0];
value = field.split('=')[1];
if value =~ /\A\d+\Z/
event.set(name, value.to_i)
else
event.set(name, value)
end
end
"
}
}
output {
stdout { codec => rubydebug }
}
Note, if you decide to upgrade to Logstash 5, there are some breaking changes:
https://www.elastic.co/guide/en/logstash/5.0/breaking-changes.html
In particular, it is the event that needs to be modified to use either event.get or event.set. Here is what I used to get it working (based on Ben Lim's example):
input {
stdin {}
}
filter {
ruby {
code => "
fieldArray = event.get('message').split(' ');
for field in fieldArray
name = field.split('=')[0];
value = field.split('=')[1];
if value =~ /\A\d+\Z/
event.set(name, value.to_i)
else
event.set(name, value)
end
end
"
}
}
output {
stdout { codec => rubydebug }
}
I'm trying display some Mongo data that I've been collecting using logstash using the Mongostat tool. It displays things with a suffix like "b", "k", "g" to signify byte, kilobyte, gigabyte, which is fine if I'm just reading the output, but I want to throw this into kibana and display it in a graphical format to see trends.
I've done this with several other log files and everything is fine. When I use a grok filter everything is fine but I've added a Ruby filter and now data seems to be duplicated in all fields other than the logstash generated fields and my new field created in my Ruby filter.
Here is the relevant parts of my conf file:
input {
file {
path => "/var/log/mongodb/mongostat.log"
type => "mongostat"
start_position => "end"
}
}
filter {
if [type] == "mongostat" {
grok {
patterns_dir => "/opt/logstash/patterns"
match => ["message","###a bunch of filtering that i know works###"]
add_tag => "mongostat"
}
if [mongoMappedQualifier] == 'b' {
ruby {
code => "event['mongoMappedKB'] = event['mongoMapped'].to_f / 1024"
}
}
if [mongoMappedQualifier] == 'k' {
ruby {
code => "event['mongoMappedKB'] = event['mongoMapped'].to_f * 1"
}
}
if [mongoMappedQualifier] == 'm' {
ruby {
code => "event['mongoMappedKB'] = event['mongoMapped'].to_f * 1024"
}
}
if [mongoMappedQualifier] == 'g' {
ruby {
code => "event['mongoMappedKB'] = event['mongoMapped'].to_f * 1048576"
}
}
}
}
output {
if [type] == "mongostat" {
redis {
host => "redis"
data_type => "list"
key => "logstash-mongostat"
}
}
}
Any idea why or how this can be fixed?