Multiline logstash "next" not grouping - elasticsearch

Short:
I am having troubles with multiline. I get the tag "multiline" on the log but it doesn't put them together.
Explanation:
Logs I receive
September 22nd 2016, 13:43:52.738 [0m[[31merror[0m] [0mTotal time: 368 s, completed 2016-09-22 13:43:52[0m
September 22nd 2016, 13:43:51.738 [0m[[0minfo[0m] [0m[36mSuites: completed 29, aborted 0[0m[0m
September 22nd 2016, 13:43:51.738 [0m[[31merror[0m] [0mFailed: Total 100,
Failed 4, Errors 0, Passed 96[0m
September 22nd 2016, 13:43:51.737 [0m[[0minfo[0m] [0m[36mRun completed in 1 minute, 24 seconds.[0m[0m
September 22nd 2016, 13:43:51.737 [0m[[0minfo[0m] [0mScalaTest[0m
The line with "Total time: %{NUMBER} s" is repeated multiple time and I'm only interested in these Total time coming after a "Total, Failed, Error" line.
Between the first and the second line could be none or several logs.
My configuration is:
grok {
#1
match => {"message" => "\[.m\[\u001b\[3.m%{NOTSPACE:level}\u001b\[0m\] \u001b\[0m%{NOTSPACE:Status}: Total %{NUMBER}, Failed %{NUMBER}, Errors %{NUMBER}$
add_tag => [ "test.continue" ]
tag_on_failure => []
}
#2
if "test.continue" in [tags]{
multiline {
pattern => "%{TIMESTAMP_ISO8601}\u001b\[0m$"
what => "next"
negate => true
}
}
#3
#OverallTime
grok {
match => {"message" => "\[.m\[\u001b\[3.m%{NOTSPACE:level}\u001b\[0m\] \u001b\[0mTotal time: %{NUMBER:Seconds:int} s, completed"}
add_tag => [ "test.overalltime" ]
tag_on_failure => []
}
What I get is:
beats_input_codec_plain_applied, test.continue, multiline [0m[[31merror[0m] [0mFailed: Total 100, Failed 4, Errors 0, Passed 96[0m
The first log gets the multiline tag and the test.continue but doesn't behave as I expect.
The logic as I understand it is:
If you find [0m[[31merror[0m] [0mFailed: Total 100, Failed 4, Errors 0, Passed 96[0m then put a tag "test.continue",
Multiline every log with the tag "test.continue" and send it to the next line you find until you find a log with the end %{TIMESTAMP_ISO8601}\u001b\[0m$
Extract the time from that log
More explanation:
I'm believing the behaviour will be, but is not happening.
1 finding the trigger
[0m[[31merror[0m] [0mFailed: Total 100,
Failed 4, Errors 0, Passed 96[0m
2 Once it finds it, it will take it and added at the beginning of the next line. Since the first part of the log will still match, it will add again the tag and then sends it to the multiline again
[0m[[31merror[0m] [0mFailed: Total 100,
Failed 4, Errors 0, Passed 96[0m [0m[[0minfo[0m] [0m[36mSuites: completed 29, aborted 0[0m[0m
3 It will have the first, second, ... , until it finds the log which finishes with a timestamp and breaks the multiline. The next log will be not added with the tag.
[0m[[31merror[0m] [0mFailed: Total 100, Failed 4, Errors 0, Passed 96[0m [0m[[0minfo[0m] [0m[36mSuites: completed 29, aborted 0[0m[0m [0m[[31merror[0m] [0mTotal time: 368 s, completed 2016-09-22 13:43:52[0m

Related

InfluxDB Measurements Have Incorrect Time

I'm having some issues with incorrect time stamps in InfluxDB. I receive data from a websocket connection, create a new point for each response, and upload each point to a bucket. When I look at the points in the InfluxDB UI, I notice that the time field is incorrect. Here are 4 examples of unmarshalled responses that I received from the websocket connection, as well as the time field in its respective point:
&types.TradeResponse{ChannelID:337, TradeArray:[]types.TradeDataResponse{types.TradeDataResponse{Price:21567.4, Volume:0.00553002, Time:time.Date(2022, time.August, 26, 3, 49, 29, 0, time.UTC), Side:"b", OrderType:"m", Misc:""}}, ChannelName:"trade", Pair:"XBT/USD"}
point time: 2022-08-26 03:49:29 +0000 UTC
&types.TradeResponse{ChannelID:337, TradeArray:[]types.TradeDataResponse{types.TradeDataResponse{Price:21567.4, Volume:0.0372093, Time:time.Date(2022, time.August, 26, 3, 49, 43, 0, time.UTC), Side:"b", OrderType:"m", Misc:""}}, ChannelName:"trade", Pair:"XBT/USD"}
point time: 2022-08-26 03:49:43 +0000 UTC
&types.TradeResponse{ChannelID:337, TradeArray:[]types.TradeDataResponse{types.TradeDataResponse{Price:21567.3, Volume:0.00045028, Time:time.Date(2022, time.August, 26, 3, 49, 59, 0, time.UTC), Side:"s", OrderType:"m", Misc:""}}, ChannelName:"trade", Pair:"XBT/USD"}
point time: 2022-08-26 03:49:59 +0000 UTC
&types.TradeResponse{ChannelID:337, TradeArray:[]types.TradeDataResponse{types.TradeDataResponse{Price:21567.3, Volume:0.00010686, Time:time.Date(2022, time.August, 26, 3, 50, 7, 0, time.UTC), Side:"s", OrderType:"m", Misc:""}}, ChannelName:"trade", Pair:"XBT/USD"}
point time: 2022-08-26 03:50:07 +0000 UTC
and here is a picture of the points in the bucket:
As you can see, the _time field is always slightly off from the time in the point. I'm currently using the go client to upload data to the DB. Here is the code that I have written for that:
func OnTradeResponse(data types.TradeResponse, tradesWriter api.WriteAPI) {
for _, trade := range data.TradeArray {
point := influxdb2.NewPoint("trade", map[string]string{"object": "trade", "pair": data.Pair}, map[string]interface{}{"price": float64(trade.Price), "volume": float64(trade.Volume), "side": trade.Side, "orderType": trade.OrderType, "misc": trade.Misc}, trade.Time.Time)
tradesWriter.WritePoint(point)
log.Println("point time: ", point.Time())
}
}
If I was just using the system time when creating new points, I could understand why there might be an issue with having an incorrect time, but since I'm using the time from the trade response as the time in the point, I don't understand what is causing the issue. Would greatly appreciate any help

Dynamic Date and Time Filter: After 5:00 PM In Previous Day

I am trying to creat a filter that will pull every account that has been set up after 5:00 PM from the previous day. The date and time exist in the same row. I have created a filter that works for the day but the next day, it pulls for two days. For example, here is what it looks like right now:
= Table.SelectRows(#"Sorted Rows", each [Driver ID] > #datetime(2021, 12, 29, 17, 0, 0))
I have tried changing it to the following so it would dynamically change as the days pass:
= Table.SelectRows(#"Sorted Rows", each DateTime.From([Driver ID]) > Date.AddDays(DateTime.From(Driver ID), -1))
But when I do this I get the following error:
Expression.Error: We cannot convert the value #datetime(2021, 12, 30, 0, 5, 0) to type Function.
Details:
Value=12/30/2021 12:05:00 AM
Type=[Type]
I have made sure the column type is in Date/Time format but that doesn’t seem to help.
Has anybody ran into this issue and know a good solution?
try
= Table.SelectRows(#"Sorted Rows", each Date.IsInPreviousNDays([DriverID], 1) and Time.From([DriverID])> #time( 17, 0, 0))

Rails split string with multiple parts

I have a string that's been imported from a csv such as:
14th Aug 2009:1, 15th Aug 2009:1, 16th Sep 2015:1|Style1, 17th Sep
2015:1|Style 1
I wish to add this data to my database in a specific way. First I split it on , to get each date group (in this case 4 dates).
Secondly i'd like a way to split each of those date groups into multiple segments. The first with the date, second with the number after the colon and then a varied amount more for each of the items separated by the | character.
Is there an decent efficient way to accomplish this in Ruby?
Looking for outcome to be a hash like so:
{ '14th Aug 2009' => 1, '15th Aug 2009' => 1, '16th Aug 2009' => 1, '16th Sep 2015' => { 1 => 'Style 1' }, '17th Sep 2015' => { 1 => 'Style 1' }
Basically if the string was like so:
15th Aug 2009:1, 16th Sep 2015:3|Style1|Style 1, 17th Sep
2015:1|Style 1
I would get
{ '15th Aug 2009' => 1, '16th Sep 2015' => { '', 'Style 1', 'Style 1' }, '17th Sep 2015' => { 1 => 'Style 1' }
Basically, the text separated by |'s should be assigned to the number after the colon. If the number is 3 and there are two sets of text after it then one is an empty string and the other two will say the text (eg: "Style 1".
Sorry for sounding very confusing.
I'm assuming that you meant for the '|' separted items to build an Array, as infused asked about. How about this?
s ="15th Aug 2009:1, 16th Sep 2015:3|Style1|Style 1, 17th Sep 2015:1|Style 1"
result = {}
s.split(',').each do |v|
date,rest = v.split(':')
items = rest.split('|')
if items[0] == "1"
result[date] = 1
else
result[date] = ['', items[1..-1]]
end
end

How to find the dates which are there in a week or month till date

How to find the dates which are there in a week or month till date.
days_for_week should return 19,20,21 (assuming current date is 21st)
days_for_month should return 1..21 (assuming current date is 21st)
For the first, you could use Time.now.wday to get the current week day, then minus that will give you the date of beginning of this week.
For the second, it's much simpler, every month begin with 1st, right?
Assuming I'm reading your question correctly...
The second is simple:
def days_for_month
1..Date.today.day
end
The first requires a little algorithm to work back to Saturday:
def days_for_week
days = []
day = Date.today
until day.saturday?
days.unshift(day.day)
day -= 1
end
days
end
Active support provides a lot of useful methods like at_beginning_of_week, at_end_of_week, at_beginning_of_month etc ..
> Date.today.at_beginning_of_week
=> Mon, 20 May 2013
For this particular case, you could do
> (Date.today.at_beginning_of_week..Date.today).map &:day
=> [20, 21]
Similarly
> (Date.today.at_beginning_of_month..Date.today).map &:day
=> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]
or simply
> 1..Date.today.day

How to find out the date of second Monday of each month of given year?

My customer has an event each second Monday of each month.
I need to mark them with red in calendar.
How do i "cleanly" find out the date of that Mondays?
Here's my version.
If the eighth of the month is a Monday, then it is the second Monday. If it is not a Monday, then how many days until the next Monday?
oct_2012 = Date.new 2012, 10, 8
oct_2012.wday # => 1, We're done!
nov_2012 = Date.new 2012, 11, 8
nov_2012.wday # => 4
nov_2012 + (8 - nov_2012.wday) # => 2012-11-12
Does that help?
Edit
Easier version: Just add and be done. This algorithm works even if the month starts on a Monday.
oct_2012 = Date.new 2012, 10, 1
oct_2012 + (8 - oct_2012.wday) # => 2012-10-08
nov_2012 = Date.new 2012, 11, 1
nov_2012 + (8 - nov_2012.wday) # => 2012-11-12
One rule and done!
You second Monday will always fall within the 8th and 14th of each month.

Resources