Graylog , Handle files before store it - bash

I am trying to create log system using Gray log,
I have logs like this
Mon Aug 17 16:53:10 2020
NAS-IP-Address = 172.20.101.1
User-Name = "netconf"
Acct-Status-Type = Start
Acct-Session-Id = "ACCTID20200817112721000fe14d000000001040717"
Acct-Authentic = RADIUS
NAS-Identifier = "NR-DELHI-NRTCC-CR-01"
Framed-IP-Address = 172.20.16.6
NAS-Port-Type = Virtual
Event-Timestamp = "Aug 17 2020 16:57:21 IST"
Tmp-String-9 = "ai:"
Acct-Unique-Session-Id = "23be63c276bbda95385d118ff93ba298"
Timestamp = 1597663390
Mon Aug 17 16:54:15 2020
NAS-IP-Address = 172.20.101.1
User-Name = "netconf"
Acct-Status-Type = Start
Acct-Session-Id = "ACCTID20200817112825000fe14f000000001040719"
Acct-Authentic = RADIUS
NAS-Identifier = "NR-DELHI-NRTCC-CR-01"
Framed-IP-Address = 172.20.16.6
NAS-Port-Type = Virtual
Event-Timestamp = "Aug 17 2020 16:58:25 IST"
Tmp-String-9 = "ai:"
Acct-Unique-Session-Id = "49fcad388a523e7eebea6043529c323e"
Timestamp = 1597663455
actually, every block is one log record, but the problem is that Graylog considering every line as a separated log, so what I want is to tell Graylog to read every block as one record.
is there any way to do that, some configuration that I have to do in Gray log to achieve my goal.
I have an idea but I am not sure about it, I am thinking about creating a bash script which will read the file and merge lines which belong to one block in one line,
but I am not sure if this method will work or not, so I am wondering if there is any way to do it from Graylog itself.
any suggestion will be appreciate
Best Regards

if messages in the log file begin with a blank line, then you could use the readmode parameter of the imfile module: https://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html#readmode

Related

How to get the same output of departed Date.parse() in groovy?

I have an application that runs the old version of the spring application. The application has the function to create date objects using Date.parse as follows
Date getCstTimeZoneDateNow() {
String dateFormat = "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"
def zonedDateString = new Date().format(dateFormat, TimeZone.getTimeZone('CST'))
Date date = Date.parse(dateFormat, zonedDateString)
return date // Tue Oct 18 20:36:12 EDT 2022 (in Date)
}
However, the code above is deprecated. I need to produce the same result.
I read other posts and it seems like Calender or SimpleDateFormatter is preferred.
And I thought SimpleDateFormatter has more capabilities.
This post helped me understand more about what is going on in the following code
SimpleDateFormat parse loses timezone
Date getCstTimeZoneDateNow() {
Date now = new Date()
String pattern = "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"
SimpleDateFormat sdf = new SimpleDateFormat()
sdf.setTimeZone(TimeZone.getTimeZone('CST'))
// cstDateTime prints times in cst
String cstDateTime = sdf.format(now) // 2022-10-18T20:36:12.088Z (in String)
// JVM current time
Date date = sdf.parse(cstDateTime) // Tue Oct 18 21:36:12 EDT 2022 (in Date)
return date
}
Here my goal is to return the date object that is in the format of Tue Oct 18 20:36:12 EDT 2022
The format is good. However, like the post says, when I do sdf.parse(), it prints in JVM time.
This means, the format is good but the time zone is off.
How can I get the exact same result as before?
It does not have to use SimpleDateFormatter. Could be anything.
Thank you so much for reading and for your time.
Perhaps the important thing is, that the Date is always neutral to the timezone. Given example shows what is to be expected to work from the Java specs:
def format = new SimpleDateFormat()
format.setTimeZone(TimeZone.getTimeZone("CST"))
println new Date()
def date = format.parse(format.format(new Date()))
printf "parsed to %s%n", date
printf "formatted to %s (%s)%n", format.format(date), format.getTimeZone().getDisplayName()
In the output, notice when using the Format and when the toString(), a different time is shown accordingly, which is perfectly fine, since first we format and then parse again in the same format, thus the same time-zone. Later, we use the Date.toString() to output the date, this time using the system default time-zone which is always used when Date.toString() is called. In the output, the time-zone shift is reflected:
Thu Oct 20 09:22:58 EDT 2022
parsed to Thu Oct 20 09:22:00 EDT 2022
formatted to 10/20/22 8:22 AM (Central Standard Time)

Compare one date value with other dates and perform conditional action in cypress

I'm trying to compare one date value (ie. base value) with all other date values on a page and based on the difference between these days, I want to execute other commands.
Well, in the above UI, the base value is 11 Jul 2021 (Departure date in the first list) and other date values are 12 Jul 2021, 20 Jul 2021, 27 Jul 2021, 3 Aug 2021 and so on (Arrival dates from 2nd list onwards).
Now, I had to delete the all list(s) where the date difference between the base value and particular list is less than 15 days.
In this case, 12 Jul 2021, 20 Jul 2021 had to be deleted and all lists from 27 Jul 2021, 3 Aug 2021 and so on should be untouched as in the below picture.
So far, I have captured the value of the base value and came up with logic to compare it with another date value but I am not sure how I can save the 2nd and further date value(s) to a variable in order to compare with the base value.
{
cy.get("[data-test='departureTime']")
.eq(0)
.then((date) => {
const depDate_FirstPort = new Date(date.text());
cy.log(depDate_FirstPort.toISOString()); //2021-07-11T19:00:00.000Z
// const arrDate_SecondPort = new Date(cy.get('[data-test="arrivalTime"]').eq(1).invoke('text'));
// Since the above approach does not work, hard coding now.
const arrDate_SecondPort = new Date("22 Jul 2021 12:01")
cy.log(arrDate_SecondPort.toISOString()); //2021-07-22T10:01:00.000Z
cy.getDifferenceBetweenDates(depDate_FirstPort,arrDate_SecondPort).then((dif)=>{
if(dif < 16) {
cy.log("delete the port entry");
//do something
}
});
});
}
Cypress Command:
Cypress.Commands.add("getDifferenceBetweenDates", (Date1, Date2) => {
var diff_times = Math.abs(Date1.getTime() - Date2.getTime());
var diff_days = Math.ceil(diff_times / (1000 * 3600 * 24));
cy.log(diff_days) //11
})
Also, curious to know a possible approach to iterate all list falls under the 'to be deleted list' (12 Jul 2021, 20 Jul 2021) based on the condition mentioned above.
The iterative approach you have is ok, but you need to repeat the code you have for the first date to get the subsequent dates.
So, this bit but changing the index
cy.get("[data-test='departureTime']")
.eq(0) // 1,2,3 etc
.then((date) => {
A different approach is to filter the whole set,
const dayjs = require('dayjs') // replaces Cypress.moment
// first install with
// yarn add -D dayjs
it('finds the skipped ports', () => {
// helper func with format specific to this website
const toDate = (el) => dayjs(el.innerText, 'D MMM YYYY HH:mm')
cy.get("[data-test='departureTime']")
.then($departures => {
const departures = [...$departures] // convert jQuery object to an array
const first = toDate(departures[0]);
const cutoff = first.add(15, 'day')
const nextPorts = departures.slice(1) // all but the first
const skipPorts = nextPorts.filter(port => toDate(port).isBefore(cutoff))
expect(skipPorts.length).to.eq(2)
expect(skipPorts[0].innerText).to.eq('12 Jul 2021 14:02')
expect(skipPorts[1].innerText).to.eq('21 Jul 2021 04:00')
})
})
I'm not clear about your goal, but if you are going to actually delete the skipPorts from the page instead of just testing them, you should be wary of the DOM list changing as you do so.
Deleting from the list you have recently queried with cy.get("[data-test='departureTime']") would cause the internal subject to become invalid, and you might get "detached from DOM" errors or delete the wrong item.

Create monthly trigger for Scheduled Task in Powershell (With additional criteria)

I'm currently working on a script that when run, creates some Scheduled tasks that makes the host machine do several things and then restart within a specified time span.
This script needs to be run on multiple domain controllers, and therefor i would like to "load balance" by using something like New-ScheduledTaskTrigger -RandomDelay in order for them to not reboot all at once, but kind of spread it out.
The goal is to be able to change some variables of when to restart, things like:
First Monday of the month between 18:00 and 23:59
Every Thursday between 01:00 and 06:00
Every day between 04:00 and ..... you see where I'm going
However there is no such thing as a "-Monthly" in New-ScheduledTaskTrigger
That's the first problem, this one i can probably solve with the help from other posts, but if i do it for example like this I'm not able to use the -RandomDelay which I think is a major feature for this to work.
Here is how I imagine it should look if the -Monthly did work (for a monthly trigger):
$rebootFrequency = MONTHLY # DAILY, WEEKLY, MONTHLY
$rebootWeek = FIRST # FIRST, SECOND, THIRD, FOURTH, LAST
$rebootDayOfWeek = MON # MON, TUE, WED, THU, FRI, SAT, SUN
$rebootTimeFrom = 10:00 # HH:MM[:SS]
$rebootTimeTo = 16:00 # HH:MM[:SS]
New-ScheduledTaskTrigger -"$rebootFrequency" -WeekOfMonth $rebootWeek;
-DayOfWeek $rebootDayOfWeek -At $rebootTimeFrom -RandomDelay $rebootTimeTo
Do you have any suggestions as to how I should solve this problem?
I could do the same thing with schtask.exe, however I would end up having to make some kind of script to do the "RandomDelay" function.
Feel free to ask further if you have any questions.
Thanks in advance.
Challenge 1
I've now got it to work, but I'm trying to make the script a bit more intuitive, but I can't figure out how i would do it...
What i want to do is to "convert" from using the numbers in days (for example: 16 for Thursday) to being able to write "THU" instead.
Right now it looks something like this:
$rebootDaysOfWeek = "16" # SUN=1, MON=2, TUE=4, WED=8, THU=16 etc.
$trigger.DaysOfWeek = $rebootDaysOfWeek
But I would find it alot cooler if it was something like this:
$rebootDaysOfWeek = "THU" # SUN, MON, TUE, WED, THU, FRI, SAT
$trigger.DaysOfWeek = $rebootDaysOfWeek
But I can't seem to find a way to "convert" $rebootDaysOfWeek to work with the bit mask.
Check out the Microsoft Docs:
https://learn.microsoft.com/en-us/windows/win32/taskschd/time-trigger-example--scripting-
The sample is in VB, but it looks like it's just a ComObject. I haven't had enough time to play around, but you can start like this:
$service = new-object -comobject Schedule.Service
$service.connect()
$taskdefinitiion = $service.NewTask(0)
There's lots of task definition stuff, but it get's down to the triggers and you'll do this:
$triggers = $taskDefinition.Triggers
$trigger = triggers.Create(5) # I had to try different numbers here, didn't dig through the docs
$trigger.DaysOfWeek = 16 #Thursday
$trigger.WeeksOfMonth = 1 # First week, 2 for second, 6 for third, 8 for forth
$trigger.MonthsOfYear = 4095 # all months
$trigger.RandomDelay = 'PT1H' # 1 hour random delay.
I'll let you take it from here. Links to some of the items above:
https://learn.microsoft.com/en-us/windows/win32/taskschd/monthlydowtrigger-daysofweek
https://learn.microsoft.com/en-us/windows/win32/taskschd/monthlydowtrigger-monthsofyear
https://learn.microsoft.com/en-us/windows/win32/taskschd/monthlydowtrigger-weeksofmonth
https://learn.microsoft.com/en-us/windows/win32/taskschd/monthlydowtrigger-randomdelay
UPDATE FOR CHALLENGE 1
In order to use "friendly" references to the bitwise decimal value you can either create a constants section or use hashtable, either way you are going to have to do the conversion yourself:
# Constants
$SUN = 1
$MON = 2
$TUE = 4
$WED = 8
$THU = 16
$FRI = 32
$SAT = 64
# Hashtable - because why not!
$DaysOfWeek = #{
SUN = 1
MON = 2
TUE = 4
WED = 8
THU = 16
FRI = 32
SAT = 64
}
Then you can use:
$trigger.DaysOfWeek = $THU
or
$trigger.DaysOfWeek = $DaysOfWeek["THU"]

How to add zero to single digit in a string

I need to to append zero in my below string whenever I get date with single digit without changing Quantity digit (below string is system generated in my application not created by user),
Data Added Quantity:1 on Dec 9 2015 modified on Jun 7 2016
I need to change this string just like below,
Data Added Quantity:125 on Dec 09 2015 modified on Jun 07 2016
So far I have tried the below regular expression, but not getting desired output.
str = "Data Added Quantity:1 on Dec 9 2015 modified on Jun 7 2016"
Set oReg = New RegExp
oReg.Pattern = "\s\d{1}\s"
Set obj = oReg.Execute(str)
For i = 0 To obj.Count-1
mD = obj.Item(i).Value
oReg.Replace(str, "0" & mD)
Next
How we can achieve this using VBScript?
If you adjust the pattern a little and set the Global option to True you can simply use the Replace method. No need to Excecute and loop.
Set re = New RegExp
re.Pattern = "\s(\d)\s"
re.Global = True
str = re.Replace(str, " 0$1 ")
\d without a modifier already matches exactly one digit, so \d{1} is redundant. The parentheses around the \d define a capturing group that allows you to use the matched substring in the replacement ($1).

(Swift) Using NSDataFormatter with a medium style string to get date

I seem to be stuck here and I have been wasting way too much tome on this.
What I have is a string that is in the RFC 1123 format that I would like to get a date out of, but not matter what I do, I get a nil result;
let dateFormat = NSDateFormatter();
dateFormat.dateStyle = .MediumStyle;
dateFormat.dateFormat = "EEE',' dd MMM yyyy HH':'mm':'ss z";
dateFormat.locale = NSLocale.systemLocale();
dateFormat.timeZone = NSTimeZone(abbreviation:"GMT");
var currentDate = dateFormat.dateFromString("Sun, 28 Jun 2015 04:30:54 GMT");
I am not sure what I am missing, if I changed the MMM to MM and make Jun 06, then it works. It seems to be only this instance. I have tried moving the order of how dateFormat gets created, and still I get no results. Any help on this matter would greatly be appreciated
I think you have confused the formatter. You don't need to set anything except the format string, because the formatter's job is to learn those other settings from the string it reads.
let dateFormat = NSDateFormatter()
dateFormat.dateFormat = "EEE',' dd MMM yyyy HH':'mm':'ss z"
var currentDate = dateFormat.dateFromString("Sun, 28 Jun 2015 04:30:54 GMT")
// "Jun 27, 2015, 11:30 PM"
If you do as above, it will return an NSDate? from the date string you provided.

Resources