I have a lists of system logs in a log file ,in which I have to calculate time difference between two particular logs based on some condition
#timestamp message
May 19th 2016, 02:55:29.003 just some log
May 19th 2016, 02:55:29.003 retired 2 times
May 19th 2016, 02:55:29.200 System1 down -------
May 19th 2016, 02:55:29.205 just some log
May 19th 2016, 02:55:29.453 System1 up ----
May 19th 2016, 02:55:39.200 System2 down
May 19th 2016, 02:55:49.205 just some log
May 19th 2016, 02:55:53.453 System2 up -------
May 19th 2016, 02:55:58.453 all done
For example in the mentioned log, we need to trigger the calculation when we see the message 'retried' in some log.
On seeing the message it should start the calculating the difference from immediately followed message which contains 'System1 Down' message till it sees the 'System1 Up' message.But here the issue is that the system message can be dynamic like either 'System1' or 'System2'
If it starts with 'System1 Up' , it should calculate time till 'System1 Down'
and same for 'System2 Up'. Is there a possibility to use regex in logstash.
And all the process should stop when it see 'all done' in the message. The time difference calculation should start again if it sees 'retired 2 times' in any of the logs that come below afterwards.
I tired to use elapsed plugin in Logstash and aggregate but not sure, it didn't work for this kinda flow.
What is best to do this? Either in Logstash , elastic search or Kibana.
Thanks in advance
Related
Assuming the following metric:
cpu_count{machine="srv1", owner="Alice", department="ops"} 8
cpu_count{machine="srv1", owner="Bob", department="ops"} 8
I'd like to be able to prevent starting a new time series on owner change. It should still be considered the same instance, but I would like to be able to look up by owner.
I don't particularly care if it matches only on my_metric{owner=~"Box"} or on both my_metric{owner=~"Box"} and my_metric{owner=~"Alice"}, I just need to make sure it does not count twice on my_metric{machine=~"srv1"} or my_metric{department=~"ops"}.
I'm willing to accept that using labels to group instances in this manner is not the correct approach, but what is?
When you add the label "owner" to this kind of metric I think you're trying to accomplish a kind of "asset management" which could be done better with some other tool developed specific to this goal. Prometheus isn't a suitable tool to keep the information of who is using each machine in your company.
Said that, every time the owner of a machine changes you could workaround this issue deleting the old data series using the REST API executing something like this:
curl --silent --user USER:PASS --globoff --request POST "https://PROMETHEUS-SERVER/api/v1/admin/tsdb/delete_series?match[]={machine='srv1',owner='Bob'}"
If you can change the code, it would be better to have a metric dedicated to the ownership:
# all metrics are identified a usual
cpu_count{machine="srv1", department="ops"} 8
# use an info metrics to give details about owner
machine_info{machine="srv1", owner="Alice", department="ops"} 1
You can still aggregate the information id you need it:
cpu_count * ON(machine,department) machine_info
That way, the owner is not polluting all your metrics. Still, you will have issues when changing the owner of a machine while waiting for the older metric to disappear (5 minutes before staleness).
I have not tried it but a solution could be to use the time at which the ownership changed (if you can provide it) as a metric value - epoch time in seconds.
# owner changed at Sun, 08 Mar 2020 22:05:53 GMT
machine_info{machine="srv1", owner="Alice", department="ops"} 1583705153
# Previous owner Sat, 01 Feb 2020 00:00:00 GMT
machine_info{machine="srv1", owner="Alice", department="ops"} 1580515200
And then use the following expression to get the latest owner whenever you need the current owner - only useful when owner has changed within the last 5 minutes:
machine_info == ON(machine,department) BOOL (max(machine_info) BY(machine,department) )
Quite a mouthful but it would approach what you want.
I am laying bets through the Betfair API and noticed something very strange. For some events the time is off on both market and event results.
My account date and time are set to Europe/London. Not that it matters, the API outputs whatever zone it wants anyway, apparently at random (sometimes GMT other times London or whatever other timezone, but the ISO timestamp does not include +0100 or whatever GMT offset), all this despite the fact that my account is set to EU/London.
For example let's take this event Superettan / Jonkopings Sodra v Varnamo
The API output is:
{"event"=>{"id"=>"28836884",
"name"=>"Jonkopings Sodra v Varnamo", "countryCode"=>"SE",
"timezone"=>"GMT", "openDate"=>"2018-08-13T18:00:00.000Z"
}, "marketCount"=>2}
Their website however says the event starts at 17:00 GMT / 18:00 London (now GMT+1).
There are hundreds examples like this. Cannot get anything out of Betfair support, they take days/weeks to reply and when they do they just send some canned message.
Has someone seen this issue before? Or perhaps I am missing something?
Later edit: I had to dump all their market data to reach this conclusion: it appears that when their API says the timezone is GMT the event is actually on London, so GMT+1 right now, and for every other timezone (Eu/London, Perth, US...) the time is actually on GMT. Not sure what's up with that. Still investigating.
All times returned by Betfair are in UTC - that's just the way the API works. They also supply time zones which may or may not be correct. Use the time zones as an indication of the local time of the event. However, the time zones are very "hit and miss", so don't rely on them to be correct. In fact, over time you will notice lots of "data errors" for non-pricing info on Betfair markets. This is just the way it goes. so don't expect them to fix the errors
Event.openDate is documented as "The scheduled start date and time of the event. This is Europe/London (GMT) by default" while Event.timezone is defined as "[the] timezone in which the event is taking place."
So, in one sense, the openDate in your API output is correct because it says 18:00. However, it's incorrect because it includes the Zulu 'Z' suffix indicating GMT / UTC. The inclusion of 'GMT' in the timezone field: a) is wrong; and b) just serves to confuse matters, IMHO.
Instead, I've started to use MarketCatalogue.marketStartTime which is defined as "The time this market starts at..." which isn't quite the same as openDate but, if they're going to be different, usually on Betfair you're more interested in when the market goes in-play than when the actual match / race / game begins.
The Events I'm using to validate this approach seem to work better this way. Bearing in mind that the UK moves from BST on 27 October to GMT on 28 October, from the API I'm receiving these as MarketCatalogue.marketStartTime:
Liverpool v Cardiff: "2018-10-27T14:00:00.000Z" [Correct: 14:00 GMT is 15:00 BST]
F1 Mexico Grand Prix [qualifying]: "2018-10-27T18:00:00.000Z" [Correct: 18:00 BST is 19:00 GMT]
Burnley v Chelsea: "2018-10-28T13:30:00.000Z" [Correct: 13:30 GMT is 13:30 GMT]
I am currently using slack, it’s a great team work tool. I have one issue though: when I create a remind say, /remind me “abc” at night, it sets the reminding time as 3pm. but this is not night, i would like to have 6pm, or better I can set it myself (i.e., set tonight as 6pm). How can I achieve this? - btw, i live in US west coast
You can use the time itself or a relative time:
/remind me "abc" at 6pm CST
/remind me "123" tomorrow night at 8pm CST
/remind me "youandme" 5 hours from now
You can also set your timezone data by exploring the /tz command
You can set it on the account level by following the instructions here.
I have a strange issue with the scheduler.
I'm sending a reporting that's always referring to the last day, however the reporting always refers to the day before that day. E.g. on the 2nd of January (time of writing this), it's the 31th of december (it has nothing to do with month / year change).
When I log in to the server, the times seem to be correct:
$ date
Do 2. Jan 08:33:48 UTC 2014
# via python:
>>> import datetime
>>> datetime.datetime.now()
datetime.datetime(2014, 1, 2, 8, 33, 44, 650541)
# the func that calcs the previous day gives the correct day as well
>>> dates.getBeginningOfYesterday().strftime('%d.%m.%Y')
'01.01.2014'
So, basically I'm not able to reproduce the error, but the scheduler keeps sending with this timegap.
My guess is a local time offset causing you to span a day boundary incorrectly. The examples you show are in UTC, but if you're using a localized timestamp for calculations there could easily be a 5+ hour time offset, causing you to be a day earlier.
I got in trouble with creation of an all day event appointment using Exchange 2010 Web Services (EWS) .
According to existing requirements to create an All day event appointment object needs to have specified start and end time (i.e. 10/20/2011 12:00:00 AM), and also timezone.
But my application converted to use EWS instead of WebDAV sets start and end time converted to GMT (Greenwich) time which then sent to Exchange server.
Such technique worked perfectly with WebDAV.
But with EWS I get weird result: appointment spans on 3 (three) days, and is NOT All day event appointment !!!
My mailbox timezone set to Pacific Standard Time (using OWA interface), and Exchange server Date and Time also set to Pacific Standard time.
Appointment start and end times are set to “2011-10-20T07:00:00.000Z” and “2011-10-21T07:00:00.000Z” respectively.
In terms of local time these times are “10/20/2011 12:00:00 AM” and “10/21/2011 12:00:00 AM” respectively (considering Daylight Saving time).
If IsAllDayEvent property of appointment object set to False – appointment created correctly – not as All day, starts at 10/20/2011 12AM and ends at 10/21/2011 12AM, and occupies only one day – October /20/2011 in Outlook Calendar.
But If isAllDayEvent property of appointment object set to True (everything rest remains the same) – appointment starts at Oct/19/2011 9:00:00PM, ends at Oct/21/2011 9:00:00PM, and is NOT All day.
It might be that I’m doing something wrong, but based on described above following question raised for me:
does EWS support Greenwich Time for All day events?
If yes – what might be my mistakes?
I appreciate any suggestion.
Sincerely
Andrew
Ran into a similar problem where my all day event was being created from 4pm the previous day to 4pm the specified date of the all day event (I'm currently in pacific standard time -8 so appears to be a UTC bug on the exchange server side).
When calling Appointment.save, use the optional second parameter, SendInvitationsMode.SendToNone, e.g.:
a.save(new FolderId(WellKnownFolderName.Calendar),
SendInvitationsMode.SendToNone);
If you prefer XML see Envelope/Body/CreateItem/#SendMeetingInvitations:
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages"
xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
<soap:Header>
<t:RequestServerVersion Version="Exchange2007"></t:RequestServerVersion>
</soap:Header>
<soap:Body>
<m:CreateItem SendMeetingInvitations="SendToNone">
<m:SavedItemFolderId>
<t:DistinguishedFolderId Id="calendar"></t:DistinguishedFolderId>
</m:SavedItemFolderId>
<m:Items>
<t:CalendarItem>
<t:Subject>From Java EWS</t:Subject>
<t:Body BodyType="HTML">the body</t:Body>
<t:Start>2014-01-03T00:00:00Z</t:Start>
<t:End>2014-01-04T00:00:00Z</t:End>
<t:IsAllDayEvent>true</t:IsAllDayEvent>
</t:CalendarItem>
</m:Items>
</m:CreateItem>
</soap:Body>
</soap:Envelope>
In addition to Pete's answer:Note that there is a difference between what Exchange has stored and what Outlook tells you. I'm writing 'pure' SOAP XML calls to an Exchange 2010 Server calendar and viewing the results through Outlook 2003. The creation calls explicitly specify UTC times and have no other time zone information. The server has UTC settings.
If I now create an allday event like this:
<mes:CreateItem SendMeetingInvitations="SendToNone">
<mes:Items>
<typ:CalendarItem>
<typ:Subject>Alldayevent</typ:Subject>
<typ:Start>2013-01-08T01:00:00.000Z</typ:Start>
<typ:End>2013-01-08T02:00:00.000Z</typ:End>
<typ:IsAllDayEvent>true</typ:IsAllDayEvent>
... Exchange correctly stores this as (GetItem output):
<t:Start>2013-01-08T00:00:00Z</t:Start>
<t:End>2013-01-09T00:00:00Z</t:End>
<t:IsAllDayEvent>true</t:IsAllDayEvent>
If Outlook is also configured for UTC this shows as an all day event for 8. January (as expected).
However, if I set Outlook to UTC+1 (Amsterdam time), the event is displayed extending over two days (and note he checkbox being blank):
Checking 'All day' in that situation results in (GetItem output):
<t:Start>2013-01-07T23:00:00Z</t:Start>
<t:End>2013-01-09T23:00:00Z</t:End>
<t:IsAllDayEvent>true</t:IsAllDayEvent>
I'm doing a DAV to EWS conversion myself. Something that might be of interest I ran across from Best Practices for Using Exchange Web Services for Calendaring Tasks (Ex 2007, but I assume applies to Exchange 2010 and 2013)
When Exchange Web Services receives a request to create a new CalendarItem for which the start and End properties are identified by non-UTC-offset strings, the server must convert the Start and End properties to Coordinated Universal Time (UTC) before the CalendarItem can be stored. The following are the rules for the conversion to UTC:
If the request contains an explicit time zone definition via a MeetingTimeZone property, the server will apply the correct offset with regard to Standard and Daylight rules as defined by the time zone.
If no explicit time zone is defined, the current time zone of the computer that is running Exchange 2007 (specifically, the Client Access server that is processing the request) will be used.
Note:
In Exchange 2007 SP1, all unspecified time zones are set to UTC instead of the time zone of the Client Access server.
Experimenting a little bit, I found that if you do not specify a timezone, EWS will indeed apply the time as UTC. If IsAllDayEvent is true start times and end are ignored besides their date component. So an all day event turns into 12:00am-12:00am UTC or 5:00pm-5:00pm on my calendar (I'm -7 UTC also). The Best Practices article recommends using the MeetingTimeZone element, but I received an error that it was depreciated, use StartTimeZone and EndTimeZone instead. Indeed adding <StartTimeZone Id="Pacific Standard Time"> seems to work.
As far as your 3 day issue goes I was able to reproduce similar results. Here is what I suspect is happening. If you tell Exchange the start time is 7am and end time is 8am, and flag it all day, it will automatically set the start and end times to yyy-mm-ddT00:00:00 and yyy-mm-dd+1T00:00:00. So if I send an appointment for 2011-11-04T07:00:00 to 2011-11-05T07:00:00 w/o the timezone element, it thinks I'm trying to span two days. The start time 2011-11-04T07:00:00 becomes 2011-11-04T00:00:00 to 2011-11-05T00:00:00. The end time 2011-11-05T07:00:00 becomes becomes 2011-11-05T00:00:00 to 2011-11-06T00:00:00. This gets thrown on the calendar as UTC. When viewed it in Outlook or in webapp it displays it in PST as Nov 3rd 5pm - Nov 6th 5pm and looks like its spanning 3 days (but only actually only 48 hours).
You need to specified MeetingTimeZone(for ES2007) or StartTimeZone(for ES2010+). I had the same problem, and it helped me.