Log File :-
LOCATION CPIC (TCP/IP) on local host with Unicode
ERROR Error Message
TIME Mon May 4 11:37:17 2020
RELEASE 721
COMPONENT CPIC (TCP/IP) with Unicode
VERSION 3
RC 473
LINE 9261
COUNTER 15
Changing trace level: handle 38331088 / destination (null) / level 0
Tried with below command to automatically get the last 10 minutes log.
sed -n "/ $(date +\%R -d "-10 min")/,$"p logfile.log | grep "ERROR"
no output displayed
Expected Output : Error message in Last 10 mins.
Any Solution ?
The challenge in this case is that the 10 minute test requires comparison of timestamp. Using pattern matching on the exact time 10 minutes ago might fail, if there was no error in the exact minutes.
As an alternative, consider the following 'AWK' based solution. It will skip lines until it see a line matching the following:
* First token is TIME
* The day of the month matching today
* Time in last 10 minutes
awk -v DD=$(date +%-d -d '-10 min') -v WHEN="$(date +\%R -d '-10 min')" '
$1 == "TIME" && $4 == +DD && $5 >= WHEN { p=1 }
p { print }
' logfile
The solution can be improved to be more generic. Current implementation will not work well during the first 10 minutes after midnight. This can be addressed (if needed) by OP.
Related
I'm trying to get the last x minutes of logs from /var/log/maillog from a remote host (I'm using this script within icinga2) but having no luck.
I have tried a few combinations of awk, sed, and grep but none have seemed to work. I thought it was an issue with double quotes vs single quotes but I played around with them and nothing helped.
host=$1
LOG_FILE=/var/log/maillog
hour_segment=$(ssh -o 'StrictHostKeyChecking=no' myUser#${host} 2>/dev/null "sed -n "/^$(date --date='10 minutes ago' '+%b %_d %H:%M')/,\$p" ${LOG_FILE}")
echo "${hour_segment}"
When running the script with bash -x, I get the following output:
bash -x ./myScript.sh host.domain
+ host=host.domain
+ readonly STATE_OK=0
+ STATE_OK=0
+ readonly STATE_WARN=1
+ STATE_WARN=1
+ LOG_FILE=/var/log/maillog
+++ date '--date=10 minutes ago' '+%b %_d %H:%M'
++ ssh -o StrictHostKeyChecking=no myUser#host.domain 'sed -n /^Jan' 8 '12:56/,$p /var/log/maillog'
+ hour_segment=
+ echo ''
Maillog log file output. I'd like $hour_segment to look like the below output also so I can apply filters to it:
head -n 5 /var/log/maillog
Jan 6 04:03:36 hostname imapd: Disconnected, ip=[ip_address], time=5
Jan 6 04:03:36 hostname postfix/smtpd[9501]: warning: unknown[ip_address]: SASL LOGIN authentication failed: authentication failure
Jan 6 04:03:37 hostname imapd: Disconnected, ip=[ip_address], time=5
Jan 6 04:03:37 hostname postfix/smtpd[7812]: warning: unknown[ip_address]: SASL LOGIN authentication failed: authentication failure
Jan 6 04:03:37 hostname postfix/smtpd[7812]: disconnect from unknown[ip_address]
Using GNU awk's time functions:
$ awk '
BEGIN {
m["Jan"]=1 # convert month abbreviations to numbers
# fill in the rest # fill in the rest of the months
m["Dec"]=12
nowy=strftime("%Y") # assume current year, deal with Dec/Jan below
nowm=strftime("%b") # get the month, see above comment
nows=strftime("%s") # current epoch time
}
{ # below we for datespec for mktime
dt=(nowm=="Jan" && $1=="Dec"?nowy-1:nowy) " " m[$1] " " $2 " " gensub(/:/," ","g",$3)
if(mktime(dt)>=nows-600) # if timestamp is less than 600 secs away
print # print it
}' file
Current year is assumed. If it's January and log has Dec we subtract one year from mktime's datespec: (nowm=="Jan" && $1=="Dec"?nowy-1:nowy). Datespec: Jan 6 04:03:37 -> 2019 1 6 04 03 37 and for comparison in epoch form: 1546740217.
Edit: As no one implemeted my specs in the comments I'll do it myself. tac outputs file in reverse and the awk prints records while they are in given time frame (t-now or future) and exits once it meets a date outside of the time frame:
$ tac file | awk -v t=600 ' # time in seconds go here
BEGIN {
m["Jan"]=1
# add more months
m["Dec"]=12
nowy=strftime("%Y")
nowm=strftime("%b")
nows=strftime("%s")
} {
dt=(nowm=="Jan" && $1=="Dec"?nowy-1:nowy) " " m[$1] " " $2 " " gensub(/:/," ","g",$3)
if(mktime(dt)<nows-t) # this changed some
exit
else
print
}'
Coming up with a robust solution that will work 100% bulletproof is very hard since we are missing the most crucial information, the year.
Imagine you want the last 10 minutes of available data on March 01 2020 at 00:05:00. This is a bit annoying since February 29 2020 exists. But in 2019, it does not.
I present here an ugly solution that only looks at the third field (the time) and I will make the following assumptions:
The log-file is sorted by time
There is at least one log every single day!
Under these conditions we can keep track of a sliding window starting from the first available time.
If you safe the following in an file extractLastLog.awk
{ t=substr($3,1,2)*3600 + substr($3,4,2)*60 + substr($3,7,2) + offset}
(t < to) { t+=86400; offset+=86400 }
{ to = t }
(NR==1) { startTime = t; startIndex = NR }
{ a[NR]=$0; b[NR]=t }
{ while ( startTime+timeSpan*60 <= t ) {
delete a[startIndex]
delete b[startIndex]
startIndex++; startTime=b[startIndex]
}
}
END { for(i=startIndex; i<=NR; ++i) print a[i] }
then you can extract the last 23 minutes in the following way:
awk -f extractLastLog.awk -v timeSpan=23 logfile.log
The second condition I gave (There is at least one log every single day!) is needed not to have messed up results. In the above code, I compute the time fairly simple, HH*3600 + MM*60 + SS + offset. But I make the statement that if the current time is smaller than the previous time, it implies we are on a different day hence we update the offset with 86400 seconds. So if you have two entries like:
Jan 09 12:01:02 xxx
Jan 10 12:01:01 xxx
it will work, but this
Jan 09 12:01:00 xxx
Jan 10 12:01:01 xxx
will not work. It will not realize the day changed. Other cases that will fail are:
Jan 08 12:01:02 xxx
Jan 10 12:01:01 xxx
as it does not know that it jumped two days. Corrections for this are not easy due to the months (all thanks to leap years).
As I said, it's ugly, but might work.
I got the following format ,
2015-04-12 11:22:04,876 - Logs - Error: OSError(16, 'Error in NAS')
2015-04-12 11:37:37,242 - Logs - Error: OSError(16, 'Error in NAS')
I want to get only last 10 mints lines when and search for "OSError(16, 'Error in NAS')"
I cant grep and for the error but couldn't implement dates and timing for grep
any advise
I used python :
with open('access.log') as f:
for line in f:
logdate = datetime.strptime(line.split(',')[0], '%Y-%m-%d %H:%M:%S')
if logdate >= datetime.now() - timedelta(minutes=10):
print(line)
To filter the lines with the desired log times you can use:
perl -mTime::Piece=:override -F, -lane 'my $now = gmtime();
print if $now - Time::Piece->strptime("$F[0]", "%Y-%m-%d %H:%M:%S")< 600' input-file
This uses Time::Piece->strptime to parse the time stamp into a usable Time::Piece object which supports comparison. By importing :override, the call to gmtime returns a Time::Piece object so that the subtraction can be made and compared to 600 (the number of seconds in 10 minutes). To further restrict output to lines which match a pattern, you can simply add && m/pattern/ to the condition, or pipe the output to a follow-on process.
I have a manually created log file of the format
date start duration description
2/5 10:00p 1:45 Did this and that.
2/6 2:00a 0:20 Woke up from my slumber.
==============================================
2:05 TOTAL time spent
There are many entries in the log. To avoid manually recomputing total time every time an entry is added, I wrote the following script:
#!/bin/bash
file=`ls | grep log`
head -n -1 $file | egrep -o [0-9]:[0-9]{2}[^ap] \
| awk '{ FS = ":" ; SUM += 60*$1 ; SUM += $2 } END { print SUM }'
First, the script assumes there is exactly one file with log in its name, and that's the file I'm after. Second, it takes all lines other than the line with the current total, greps the time information from the line, and feeds it to awk, which converts it to minutes.
This is where I run into problems. The final sum would always be slightly off. Through trial and error, I discovered that awk will never count the second field of the very first record, e.g. the 45 minutes in this case. It will count the hour; it won't count the minutes. It has no such problem with the other records, but it's always off by the minutes in the first record.
What could be causing this behavior? How do I debug it?
You set FS in the loop and it's already too late for the first line.
The right way to do is :
echo -e "1:45\n0:20" | awk 'BEGIN { FS=":" } { SUM += 60*$1 + $2 } END { print SUM }'
You did not show us, that how you expect output
Whether like this ?
$ cat log
date start duration description
2/5 10:00p 1:45 Did this and that.
2/6 2:00a 0:20 Woke up from my slumber.
==============================================
2:05 TOTAL time spent
Awk Code
awk '$3~/([[:digit:]]):([[:digit:]])/ && !/TOTAL/{
split($3,A,":")
sum+=A[1]*60+A[2]
}
END{
print "Total",sum,"Minutes"
}' log
Resulting
Total 125 Minutes
I'm sure this answer is obvious but I'm banging my head on it and getting a headache and my Search Foo is failing me…
I have a log file with this date format:
Sep 1 16:55:00 stuff happening
Sep 1 16:55:01 THIS IS THE LINE YOU WANT at this time stamp
Sep 1 16:55:02 more stuff
Sep 1 16:55:02 THIS IS THE LINE YOU WANT at this time stamp
Sep 1 16:55:03 blah
Sep 1 16:55:04 blah and so on…..
My ultimate goal is to:
Find the last line in the log file with a given string eg: "THIS IS THE LINE…" this is my "magic time" that I will do calculations on later.
Take the date of that line and set a variable that is the date +NN seconds. The time in the future will usually just short of 24hrs in the future from the time in step 1 so crossing into the next day may happen if that is important.
At some point in the script, advance the system clock to the new date/time after which I will be checking for certain events to fire.
I know this is way wrong but so far I have figured out how to:
Grab the last date stamp for my event.
logDate=cat /logdir/my.log | grep "THIS IS THE LINE" | tail -1 | cut -f1,2,3 -d" "
Returns: Sept 1 16:55:02
Convert the date into a more usable format
logDate2="$(date -d "$logDate" +"%m-%d %H:%M:%S")"; echo $logDate2
Returns: 09-17 16:55:02
I'm stuck here - what I want is:
futuredate=$logdate2 + XXXSeconds
Could someone help me with the time calculation or perhaps point out a better way to do all of this?
Thanks.
I'm stuck here - what I want is:
futuredate=$logdate2 + XXXSeconds
You can do it by converting through timestamps:
# convert log date to timestamp
logts="$(date -d "$logDate" '+%s')"
# add timestamp with seconds
futurets=$(( logts + XXXSeconds ))
# get date based from timestamp, optionally you can add a format.
futuredate=$(date -d "#${futurets}")
# Get time in seconds from the epoc (1970-01-01 00:00:00 UTC)
dateinseconds=$(date +"%s" -d "$(tail -1 logfile | grep "THIS IS THE LINE" | awk '{print $1, $2, $3}')")
# You can also use just awk without grep and tail to match and print the last line
dateinseconds=$(date +"%s" -d "$(awk '{/THIS IS THE LINE/}END{print $1, $2, $3}' logfile)")
gotofuture=$(( $dateinseconds + 2345 )) # Add 2345 seconds
newdate=$(date -d "#${gotofuture}")
echo "$newdate"
Trying to pull the last 5 minutes of logs with (grep matches)
so i do a tac syslog.log | sed / date -d "5 minutes ago"
every line on the log shows this format
Jun 14 14:03:58
Jul 3 08:04:35
so i really want to get the check of data from
Jul 4 08:12
Jul 4 08:17
i tried this method but KINDA works (though its still going through every day from this that 08:12: through 08:17: fits in)
e=""
for (( i = 5; i >= 0; i-- ))
do
e='-e /'`date +\%R -d "-$i min"`':/p '$e;
done
tac /var/log/syslog.log | sed -n $e
e=""
for (( i = 5; i >= 0; i-- ))
do
if [[ -z $e ]]
then e=`date +\%R -d "-$i min"`
else e=$e'\|'`date +\%R -d "-$i min"`
fi
done
re=' \('$e'\):'
tac /var/log/syslog.log | sed -n -e "/$re/p" -e "/$re/!q"
This creates a single regular expression listing all the times from the last 5 minutes, connected with \|. It prints the lines that matches them. Then it uses the ! modifier to quit on the first line that doesn't match the RE.
If you know the format of the dates then why not do:
tac syslog.log | awk '/Jul 4 08:17/,/Jul 4 08:12/ { print } /Jul 4 08:11/ {exit}'
/ .. /,/ .. / is regex range. It will print everything in this range. So as soon as you see /Jul 4 08:11/ on your line that would mean your 5 minutes window has been captured, you exit perusing the file.
So it didnt really work for the above method But i think i got it to work
if i see this i added a RANGE for the {exit}
awk '/'"$dtnow"'/,/'"$dt6min"'/ { print } /'"$dt7min"'/,/'"$dt11min"'/ {exit}'
Seems to work im testing it again
OK Finally looks like it really works this time (where it exits after the hour using SED instead of awk finally got it to work running through some tests.
tac /var/log/syslog.log | sed -e "$( date -d '-1 hour -6 minutes' '+/^%b %e %H:/q;'
date -d '-1 day -6 minutes' '+/^%b %e /q;'
date -d '-1 month -6 minutes' '+/^%b /q;'
for ((o=0;o<=5;o++)) do date -d "-$o minutes" '+/^%b %e %R:/p;'; done ; echo d)"
It works if log entries begins from "May 14 11:41". Variable LASTMINUTES is used to set the last n minutes in the log:
cat log | awk 'BEGIN{ LASTMINUTES=30; for (L=0;L<=LASTMINUTES;L++) TAB[strftime("%b %d %H:%M",systime()-L*60)] } { if (substr($0,0,12) in TAB) print $0 }'
To run the above script you need gawk which can be installed by:
apt-get install gawk
or
yum install gawk