Logstash replacing #timestamp - filter

I am using logstash to keep track of my logs.
I use file as input and elasticsearch as output.
my config file looks like this:
input{
file{
path => "C:\products.logs"
format => "json"
}
}
filter{
}
output{
elasticsearch{
host => localhost
}
}
the file is filled with lines of logs and each log looks something like this:
{"ID":65464,"Name":"Tracker_56213453.xml","sender_sent_time":"10/04/2014 14:14:40","insertion_time":"10/04/2014 14:14:40","Is_Valid":true}
as you can see, there is more then 1 value which holds time format.
when I tried to add this date filter:
filter{
date{
match=>["insertion_time","dd/MM/YYYY HH:mm:ss"]
}
}
but it didn't work.
what am I doing wrong?
thanks.


The default target for date is #timestamp. If you want to convert in place, you'll want to add a target => insertion_time


Your filter should look like the following:
filter{
date{
match=>["insertion_time","dd/MM/YYYY HH:mm:ss"]
locale => "en"
timezone => "UTC"
target => ["insertion_time"]
}
}


use this:
filter {
# match date and time
date {
"match" => ["insertion_time", "dd/MM/YYYY HH:mm:ss"]
"target" => "insertion_time"
}
}

Related

Logstash parses logfiles to wrong index

I have an elasticsearch instance, which parses my logfiles according to my regex pattern, which takes the date from the logfile. Then the date should be used as the index pattern for elasticsearch purposes. This is where it gets wrong. My logstash-pipeline-config-file looks as follows:
input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{LOGGERLEVEL:log}%{PIPE:k}%{TIMESTAMP_ISO8601:datetime}%{GREEDYDATA:data}"}
}
date {
match => ["datetime", "ISO8601"]
timezone => "Europe/Helsinki"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
}
LOGGERLEVEL and PIPE are user defined regexes. This version parses the logs as it should but it indexes the first two hours of the day to the date before. If I change the config-file as follows, elasticsearch will ignore the first two hours altogether:
input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{LOGGERLEVEL:log}%{PIPE:k}%{TIMESTAMP_ISO8601:datetime}%{GREEDYDATA:data}"}
}
date {
match => ["datetime", "ISO8601"]
timezone => "Europe/London"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
}
How should I configurate my pipeline that one day would be indexed as a whole and under the same index that is the date found from the logfile?

Solved the problem. Using kibana, go to management -> advanced settings and change the dateFormat:tz to the desired format. In my case I have to use the second configuration and select Europe/London from kibana-settings.

Logstash - Setting a timestamp from a JSON parsed object

I am having an issue with setting a timestamp from a JSON parse.
I have this string:
[{"orderNumber":"423523-4325-3212-4235-463a72e76fe8","externalOrderNumber":"reactivate_22d6ff0d8f55eb821be14df9d35505a6","operation":{"name":"CAPTURE","amount":134,"status":"SUCCESS","createdAt":"2015-05-11T09:14:30.969Z","updatedAt":{}}}]
I parse it as a json using this Logstash filter:
grok {
match => { "message" => "\[%{GREEDYDATA:firstjson}\]%{SPACE} \[%{GREEDYDATA:secondjson}\}]}]"}
}
json{
source => "firstjson"
}
date {
match => [ "operation.createdAt", "ISO8601"]
}
mutate {
remove_field => [ "firstjson", "secondjson" ]
}
}
This creates a document inside the ElasticSearch. I have a field named operation.createdAt which is properly recognised as a date field. But for some reason, this line:
date {
match => [ "operation.createdAt", "ISO8601"]
}
is not setting #timestamp field. Current #timestamp field is set at the moment of document insertion. What am I doing wrong?

Thanks to nice people at ES Logstash Community, I have found the answer.
Instead of:
date {
match => [ "operation.createdAt", "ISO8601"]
}
I use this:
date {
match => [ "[operation][createdAt]", "ISO8601"]
}
and that properly extracts and parses the JSON time object.

Extract Parameter (sub-string) from URL GROK Pattern

I have ELK running for log analysis. I have everything working. There are just a few tweaks I would like to make. To all the ES/ELK Gods in stackoverflow, I'd appreciate any help on this. I'd gladly buy you a cup of coffee! :D
Example:
URL: /origina-www.domain.com/this/is/a/path?page=2
First I would like to get the entire path as seen above.
Second, I would like to get just the path before the parameter: /origina-www.domain.com/this/is/a/path
Third, I would like to get just the parameter: ?page=2
Fourth, I would like to make the timestamp on the logfile be the main time stamp on kibana. Currently, the timestamp kibana is showing is the date and time the ES was processed.
This is what a sample entry looks like:
2016-10-19 23:57:32 192.168.0.1 GET /origin-www.example.com/url 200 1144 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-" "-"
Here's my config:
if [type] == "syslog" {
grok {
match => ["message", "%{IP:client}\s+%{WORD:method}\s+%{URIPATHPARAM:request}\s+%{NUMBER:bytes}\s+%{NUMBER:duration}\s+%{USER-AGENT}\s+%{QS:referrer}\s+%{QS:agent}%{GREEDYDATA}"]
}
date {
match => [ "timestamp", "MMM dd, yyyy HH:mm:ss a" ]
locale => "en"
}
}
ES Version: 5.0.1
Logstash Version: 5.0
Kibana: 5.0
UPDATE: I was actually able to solve it by using:
grok {
match => ["message", "%{IP:client}\s+%{WORD:method}\s+%{URIPATHPARAM:request}\s+%{NUMBER:bytes}\s+%{NUMBER:duration}\s+%{USER-AGENT}\s+%{QS:referrer}\s+%{QS:agent}%{GREEDYDATA}"]
}
grok {
match => [ "request", "%{GREEDYDATA:uri_path}\?%{GREEDYDATA:uri_query}" ]
}
kv {
source => "uri_query"
field_split => "&"
target => "query"
}

In order to use the actual timestamp of your log entry rather than the indexed time, you could use the date and mutate plugins as such to override the existing timestamp value. You could have your logstash filter look, something like this:
//filtering your log file
grok {
patterns_dir => ["/pathto/patterns"] <--- you could have a pattern file with such expression LOGTIMESTAMP %{YEAR}%{MONTHNUM}%{MONTHDAY} %{TIME} if you have to change the timestamp format.
match => { "message" => "^%{LOGTIMESTAMP:logtimestamp}%{GREEDYDATA}" }
}
//overriding the existing timestamp with the new field logtimestamp
mutate {
add_field => { "timestamp" => "%{logtimestamp}" }
remove_field => ["logtimestamp"]
}
//inserting the timestamp as UTC
date {
match => [ "timestamp" , "ISO8601" , "yyyyMMdd HH:mm:ss.SSS" ]
target => "timestamp"
locale => "en"
timezone => "UTC"
}
You could follow up Question for more as well. Hope it helps.

grok {
match => ["message", "%{IP:client}\s+%{WORD:method}\s+%{URIPATHPARAM:request}\s+%{NUMBER:bytes}\s+%{NUMBER:duration}\s+%{USER-AGENT}\s+%{QS:referrer}\s+%{QS:agent}%{GREEDYDATA}"]
}
grok {
match => [ "request", "%{GREEDYDATA:uri_path}\?%{GREEDYDATA:uri_query}" ]
}
kv {
source => "uri_query"
field_split => "&"
target => "query"
}

Drop log messages containing a specific string

So I have log messages of the format :
[INFO] <blah.blah> 2016-06-27 21:41:38,263 some text
[INFO] <blah.blah> 2016-06-28 18:41:38,262 some other text
Now I want to drop all logs that does not contain a specific string "xyz" and keep all the rest. I also want to index timestamp.
grokdebug is not helping much.
This is my attempt :
input {
file {
path => "/Users/username/Desktop/validateLogconf/logs/*"
start_position => "beginning"
}
}
filter {
grok {
match => {
"message" => '%{SYSLOG5424SD:loglevel} <%{JAVACLASS:job}> %{GREEDYDATA:content}'
}
}
date {
match => [ "Date", "YYYY-mm-dd HH:mm:ss" ]
locale => en
}
}
output {
stdout {
codec => plain {
charset => "ISO-8859-1"
}
}
elasticsearch {
hosts => "http://localhost:9201"
index => "hello"
}
}
I am new to grok so patterns above might not be making sense. Please help.

To drop the message that does not contain the string xyz:
if ([message] !~ "xyz") {
drop { }
}
Your grok pattern is not grabbing the date part of your logs.
Once you have a field from your grok pattern containing the date, you can invoque the date filter on this field.
So your grok filter should look like this:
grok {
match => {
"message" => '%{SYSLOG5424SD:loglevel} <%{JAVACLASS:job}> %{TIMESTAMP_ISO8601:Date} %{GREEDYDATA:content}'
}
}
I added a part to grab the date, which will be in the field Date. Then you can use the date filter:
date {
match => [ "Date", "YYYY-mm-dd HH:mm:ss,SSS" ]
locale => en
}
I added the ,SSS so that the format match the one from the Date field.
The parsed date will be stored in the #timestamp field, unless specified differently with the target parameter.

to check if your message contains a substring, you can do:
if [message] =~ "a" {
mutate {
add_field => { "hello" => "world" }
}
}
So in your case you can use the if to invoke the drop{} filter, or you can wrap your output plugin in it.
To parse a date and write it back to your timestamp field, you can use something like this:
date {
locale => "en"
match => ["timestamp", "ISO8601"]
timezone => "UTC"
target => "#timestamp"
add_field => { "debug" => "timestampMatched"}
}
This matches my timestamp in:
Source field: "timestamp" (see match)
Format is "ISO...", you can use a custom format that matches your timestamp
timezone - self explanatory
target - write it back into the event's "#timestamp" field
Add a debug field to check that it has been matched correctly
Hope that helps,
Artur

Parsing a string as date in logstash while inserting in elasticsearch

One record in my csv file looks like
,No,FMN1116CD,Holiday Order,2782427,Mr Vijay Ran ,9/6/2014,17/11/2014,,To Deliver,0,S,FALSE,2726149,-1,-1,,0,,,,-1,Delhi Inbound,NEW DELHI,Basic Hotel Order Details,Custom Package,Double,,Others,25500,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,890,0,0,3280,0,29670,,,FALSE,not paid,287894747,,TRUE,,5,-301767,0,50764,,,,,,,,,,,,-3065174,,,,,,,,,,,,,,,,,,,,2,India,22/11/2014,,Manual,Custom Package,26889,Balasore,Callcenter,0,0,0,0
My conf file looks like
input {
file {
path => "/home/sagnik/Work/logstash-1.4.2/bin/ho.csv"
start_position => "beginning"
}
}
filter {
date {
match => ["Travel_Date", "dd/MM/YYYY"]
}
csv {
columns => ["Comm_Plan","Queue_Booking","Order_Reference","Multi_Ordertype","Order_Item_Id","Pax_Name","Generation_Date","Travel_Date","Desk_ID","Status","SalesID","UserRole","Group_Booking","Agent_ID","Admin_ID","Partner_ID","Partner_Name","AgencyAdmin_Id","Supp_Pmt_Ref","Supp_Pmt_Acc","Supp_Pmt_Status","Distributor","Agent_Name","State","Supplier_Code","Secondary_Supplier_Code","Supplier_Number","PNR","Ticket_Number","Basic","Taxes","OCTax","Meal_Price","Cab_Price","Handling","PLB","Deposit_Incentive","Subagent_Handling","Subagent_Plb","Subagent_Deposit_Incentive","Dist_Comm","Stax_Air","Booking_Surcharge","TDS","SubAgent_TDS","Dist_TDS","Dist_Service_Tax","STax_Bas","Partner_Booking_Fee","Old_Payment_Fee","Transaction_Fee_Rcvd","Transaction_Fee_Givn","Net_Amount","Vouchers","CC","Dist_Credit","Partner_Payment_Status","Call_CenterId","Linked_Order","Is_Holiday","Child_Ordertype","Room_Nights","Payment_Sum","Credit_Outstanding","Payment_Fee","DepositCharge","DepositComm_Cr","CreditCharge","CreditComm_Cr","Distributor_CreditCharge","Distributor_CreditComm_Cr","Vendor_7Charge","CCICICI_MOTO_3DCharge","IPSPCharge","NetBanking_TPSCharge","CCICICI_EMICharge","NetBanking_CITRUSCharge","CCHDFC_MOTOCharge","ACharge","CCAMEXCharge","NetBanking_4Charge","NetBanking_PayUCharge","Ccivrscharge","Vch_LossVoucher","Vch_StaffTravel","Vch_DiscountB2C","Vch_ViaPointsRedemption","Vch_DealVoucher","Vch_BonusRedemption","Vch_Loss","Vch_MultiOrder","Vch_SME","Vch_TripCard","Vch_NetPayments","Vch_OfferPromo","Vch_HotelPromotion","No_Of_Pax","Hotel_CountryName","Checkout_Date","Hotel_Booking_Code","Hotel_Type","Hotel_Name","Hotel_Id","Hotel_City","Hotel_Booked_By","Hotel_Net","Hotel_Taxes","Hotel_Gross","Hotel_Supplier_Commission"]
separator => ","
}
}
output {
elasticsearch {
host => "localhost"
index => "hotel"
}
stdout { codec => rubydebug }
}
But after insertion the Travel_Date is coming as a string and not a date. As a result I am unable to do any navigation with it. Please help

you need to use target for this case
date {
match => ["Travel_Date", "dd/MM/YYYY"]
target => "New_Field_Name"
}

I think you have misunderstanding date plugin. date plugin is used to parse a field value and match it to #timestamp field.
If you need convert a field value from string to date, you can use Ruby plugin to do it.
With the below conf I can parse the Travel_Date to date format and navigate it in elasticsearch.
Have a look.
input {
file {
path => "/home/sagnik/Work/logstash-1.4.2/bin/ho.csv"
start_position => "beginning"
}
}
filter {
csv {
columns => ["Comm_Plan","Queue_Booking","Order_Reference","Multi_Ordertype","Order_Item_Id","Pax_Name","Generation_Date","Travel_Date","Desk_ID","Status","SalesID","UserRole","Group_Booking","Agent_ID","Admin_ID","Partner_ID","Partner_Name","AgencyAdmin_Id","Supp_Pmt_Ref","Supp_Pmt_Acc","Supp_Pmt_Status","Distributor","Agent_Name","State","Supplier_Code","Secondary_Supplier_Code","Supplier_Number","PNR","Ticket_Number","Basic","Taxes","OCTax","Meal_Price","Cab_Price","Handling","PLB","Deposit_Incentive","Subagent_Handling","Subagent_Plb","Subagent_Deposit_Incentive","Dist_Comm","Stax_Air","Booking_Surcharge","TDS","SubAgent_TDS","Dist_TDS","Dist_Service_Tax","STax_Bas","Partner_Booking_Fee","Old_Payment_Fee","Transaction_Fee_Rcvd","Transaction_Fee_Givn","Net_Amount","Vouchers","CC","Dist_Credit","Partner_Payment_Status","Call_CenterId","Linked_Order","Is_Holiday","Child_Ordertype","Room_Nights","Payment_Sum","Credit_Outstanding","Payment_Fee","DepositCharge","DepositComm_Cr","CreditCharge","CreditComm_Cr","Distributor_CreditCharge","Distributor_CreditComm_Cr","Vendor_7Charge","CCICICI_MOTO_3DCharge","IPSPCharge","NetBanking_TPSCharge","CCICICI_EMICharge","NetBanking_CITRUSCharge","CCHDFC_MOTOCharge","ACharge","CCAMEXCharge","NetBanking_4Charge","NetBanking_PayUCharge","Ccivrscharge","Vch_LossVoucher","Vch_StaffTravel","Vch_DiscountB2C","Vch_ViaPointsRedemption","Vch_DealVoucher","Vch_BonusRedemption","Vch_Loss","Vch_MultiOrder","Vch_SME","Vch_TripCard","Vch_NetPayments","Vch_OfferPromo","Vch_HotelPromotion","No_Of_Pax","Hotel_CountryName","Checkout_Date","Hotel_Booking_Code","Hotel_Type","Hotel_Name","Hotel_Id","Hotel_City","Hotel_Booked_By","Hotel_Net","Hotel_Taxes","Hotel_Gross","Hotel_Supplier_Commission"]
separator => ","
}
ruby {
code => "
event['Travel_Date'] = Date.parse(event['Travel_Date']);
"
}
}
output {
elasticsearch {
host => "localhost"
index => "hotel"
}
stdout { codec => rubydebug }
}
Hope this can help you.

Resources