I have a text log file, the format is like the following
Thread-28689296: Thu Aug 25 15:18:41 2016 [ info ]: xxxxx xxxxxx xxxxx
So I want to run cron job to find some certain error messages in last a few minutes. I wrote the following command
awk -vDate=`date +%b %d %H:%M:%S %Y` -vDate2=`date --date="2 minutes ago" +%b %d %H:%M:%S %Y` '$5 > Date && $5 < Date2' /var/log/dummy.log | grep "Fatal"
In the above command, i search for messages that have a timestamp beween time now and 2 minutes ago with a string Fatal.
But I got the following error
date: extra operand %d'
Try date --help' for more information.
date: extra operand %d'
Try date --help' for more information.
If I run date commands, I got the results as the following
date "+%b %d %H:%M:%S %Y"
Aug 25 15:25:01 2016
date --date="2 minutes ago" +"%b %d %H:%M:%S %Y"
Aug 25 15:31:42 2016
So the date commands in my awk script should be okay.
I also want to redirect the found error messages happening 2 minutes to a file to mail as alert but I did not get that far yet.
Please kindly advice me what is wrong in my awk script. Thanks a lot in advance!
The problem here is with date itself. Let's see how.
You are saying:
vDate2=`date --date="2 minutes ago" +%b %d %H:%M:%S %Y`
Because you want to use
date --date="2 minutes ago" +%b %d %H:%M:%S %Y
However, if you try to run it you'll see that you get the error:
date: extra operand ‘%d’
Try 'date --help' for more information.
The problem is that you need to enclose the FORMAT controls within double quotes:
# v v
$ date --date="2 minutes ago" "+%b %d %H:%M:%S %Y"
Aug 25 14:49:31 2016
When this is done, all together your full awk one-liner can be:
awk -v Date="$(date "+%b %d %H:%M:%S %Y")" \
-v Date2="$(date --date="2 minutes ago" "+%b %d %H:%M:%S %Y")" \
'$5 > Date && $5 < Date2' file
Note I am using -v Date="$(date ...)":
$( ) for process substitution, since backticks ` are almost deprecated, ir at least considered legacy.
date=" things " to prevent errors if the content has spaces.
v var=value using spaces after -v, since -vvar=value is gawk-specific.
Related
I want to modify this command to subtract 30 days from current date automatically
$ awk -v t=$(date +%Y-%m-%d) -F "'" '$1 < t' myname.dat
When I try
$ awk -v t=$(date "--date=$(date) -30days" +%Y-%m-%d) -F "'" '$1 < t' myname.dat
I get the following error; date: illegal option
I want to do this without having to convert the dates to epoch time in the file.
#edit: The following will work with GNU date only:
You can always subtract seconds.
date --date=#$(($(date +%s) - 30 * 24 * 3600)) +%Y-%m-%d
If you are interested in subtracting 30 days form "now", just:
date --date="-30days" +%Y-%m-%d
date date formatting is so broad, it's good to specify the exact date with for example -I option, from man date:
-I[FMT], --iso-8601[=FMT]
output date/time in ISO 8601 format. FMT='date' for date only
(the default), 'hours', 'minutes', 'seconds', or 'ns' for date
and time to the indicated precision. Example:
2006-08-14T02:34:56-06:00
The following:
date --date="$(date -I) -30days" +%Y-%m-%d
works on my system as expected.
my log has this date format at the beginning of each line:
2018 Sep 21 17:16:27:796
I need to grep the last 10 minutes of this log... any help?
my current experiments:
tenminutesago=$(date --date='10 minutes ago' +"%Y %b %e %H:%M:%S"):999
My idea was to convert the log format to a progressive number and then check everything greater than that number.
I see that the command: date +"%Y %b %e %H:%M:%S" gives a date in the same format of the log. The command: date +"%Y%m%e%H%M%S" gives a date in a progressive number (201810041204019)
You could do
for i in {10..0}; do
d=$(date -d "$i minutes ago" +'%Y %b %e %H:%M')
grep "$d" logfile
done
This just divides the problem in the 11 sequential subtasks of getting all lines from 10 minutes ago, all lines from 9 minutes ago, etc. until the current minute.
Edit:
Here's an alternate solution that prints all lines following the first one where a date stamp from the last 10 minutes was found, not only those that carry a date stamp, and also avoids reading the file over from start several times:
# build a regex pattern that matches any date in the given format from the last 10 minutes
pattern=$(date +'%Y %b %e %H:%M')
for i in {10..1}; do
pattern+=\|$(date -d "$i minutes ago" +'%Y %b %e %H:%M')
done
# print all lines starting from the first one that matches one of the dates in the pattern
awk "/$pattern/,0" logfile
Under the assumption that your loglines looks like
YYYY Bbb dd HH:MM:SS:sss Some random log message is here
You can do the following:
awk -v d=$(date -d "10 minutes ago" "+%Y %m %d %T") '
{ mm = sprintf("%0.2d",(index("JanFebMarAprMayJunJulAugSepOctNovDec",$2)+2)/3)
s = $1 " " mm " " $3 " "$4 }
(s >= d){print}' logfile
The idea is to convert your date format into a Sortable format (Note that "Jan" < "Mar" but "Feb" < "Jan"). This is done by converting your month into a number with two digits and then compare it stringwise against the correct date.
Try your current approach without the seconds and milliseconds.
tenminutesago=$(date --date='10 minutes ago' +"%Y %b %e %H:%M")
Is not exactly the last ten minutes to a second level, but I think it is enough for most of the cases. That will give you the first line in the log within the time window. Now you can get the total lines and subtract the line number of your previous grep, and then tail the file. The script could be like this:
LOGFILE="filename.log"
tenminutesago=$(date --date='9000 minutes ago' +"%Y %b %e %H:%M") # matching pattern
tlines=$(cat $LOGFILE | wc -l) # Total lines in file
let lines=$tlines-$(grep -n "$tenminutesago" $LOGFILE | grep -m 1 -oP "^[0-9]*" || echo $tlines) # lines after matching occurence
echo "$lines lines FOUND from the last X minutes"
tail -n $lines $LOGFILE # last lines in file
As suggested by #Gem Taylor, this could be reduced using +N option in tail.
LOGFILE="filename.log"
tenminutesago=$(date --date='9000 minutes ago' +"%Y %b %e %H:%M") # matching pattern
lines=$(grep -n "$tenminutesago" $LOGFILE | grep -m 1 -oP "^[0-9]*" || echo "0") # lines after matching occurence
echo "$lines lines FOUND from the last X minutes"
let lines -eq 0 && tail -n +$lines $LOGFILE # last lines in file if lines is not 0
I have an application using its own log file format. Now I want to get all the lines of logs with certain string values such as "Fatal error" within a certain period of time. The log data format is like the following:
Thread-28689296: Thu Aug 25 15:18:41 2016 [ info ]: abcd efddf
Thread-28689296: Thu Aug 25 15:19:01 2016 [ info ]: xvbdfdre dfdfd
Thread-28689296: Thu Aug 25 15:19:11 2016 [ info ]: Fatal error
Thread-28689296: Thu Aug 25 15:19:41 2016 [ info ]: dfdfdfd
If "now" is Aug 25 15:19:41 2016, I want to find between 15:19:41 and 15:17:41 those lines that have "Fatal error" in my log file. So the current time should be from date and x minutes ago should be from "date x minutes ago" to find certain error messages from the application log.
If I use the following command line:
awk -v Date="$(date "+%b %d %H:%M:%S %Y")" -v Date2="$(date -- date="2
minutes ago" "+%b %d %H:%M:%S %Y")" '$5 > Date && $5 < Date2' log_file |
grep "Fatal error"
the variable "$5" in the condition actually gets the value of minute "17" and "19" in my sample log data but it compares with a date value. So this won't work.
How can I construct the value of time in log timestamp from $3 to $7 fields to compare the value of current time. I m not so familiar with shell scripting.
Thanks for your advice and help in advance.
Can you try this:
#!/bin/sh
Date="$(date "+%b %d %H:%M:%S %Y")"
Date2="$(date --date="2 minutes ago" "+%b %d %H:%M:%S %Y")"
first_line=$(grep -n "$Date2" log_file | awk -F ":" '{print $1}')
last_line=$(grep -n "$Date" log_file | awk -F ":" '{print $1}')
sed -n "${first_line},${last_line}p" log_file | grep "Fatal error"
One in Gnu AWK:
$ cat > check.awk
BEGIN {
start=mktime("2016 8 25 15 19 00") # desired start time. If now, use something like:
# end=strftime("%s"); start=end-120
end=mktime("2016 8 25 15 19 12") # end time for this example
# make text months into numbers:
split("Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec",mons,",")
for(i in mons)
mons[mons[i]]=i
}
{
atime=mktime($6" "mons[$3]" "$4" "gensub(/:/," ","g",$5)) # your log time format
}
atime<end && atime>start # && add your desired error messages here
$ awk -f check.awk yer.log
Thread-28689296: Thu Aug 25 15:19:01 2016 [ info ]: xvbdfdre dfdfd
Thread-28689296: Thu Aug 25 15:19:11 2016 [ info ]: Fatal error
In nginx access.log we have time format in GMT timezone. using awk command,converting logs to csv file.
awk '{print $4","$7,"$10","$15","$16","$17}' </usr/local/nginx/logs/access.log>access.csv
$4 - displays date in GMT (31/Jul/2015:16:03:07).
Kindly guide how to change it to IST and update in that csv file.
If you have access to GNU awk and GNU date, you could use TZ to easily get time at IST:
$ TZ='Asia/Kolkata' date -d "#1438374787"
Where the '1438374787' value is the seconds since epoch in GMT, also known as systime.
To make a systime out of a date, we could use mktime from (GNU) awk:
Convert the GMT string to epoch time directly in awk:
$ echo "a test date 31/07/2015:16:33:07" | awk '{gsub("[/:]"," ",$4); $0=$0;t=sprintf("%d %d %d %d %d %d 0",$6,$5,$4,$7,$8,$9);print(mktime(t))}'
For that to work, you need all values to be numbers (no jul, sorry). If that is not possible, you need to work out a format for an string that the command date could read correctly (sometimes not an easy task).
Placing all in one script:
#!/bin/bash --
echo "a test date 31/07/2015:16:33:07" | awk '{
split($4,A,"[:/]");
gmtdate=sprintf("%d %d %d %d %d %d",A[3],A[2],A[1],A[4],A[5],A[6]);
print gmtdate;
T1=mktime(gmtdate);
print T1;
pt = sprintf ("TZ=%s%s%s date -d %s%s%s%s","\047","Asia/Kolkata","\047","\042","#",T1,"\042");
print pt;
system(pt);
}'
And, running it:
$ ./stackoverflow-awk-time-test.sh
2015 7 31 16 33 7
1438374787
TZ='Asia/Kolkata' date -d "#1438374787"
Sat Aug 1 02:03:07 IST 2015
Several additional debug values are printed from awk (easy to remove).
How would you parse a date in bash, with separate fields (years, months, days, hours, minutes, seconds) into different variables?
The date format is: YYYY-MM-DD hh:mm:ss
Does it have to be bash? You can use the GNU coreutils /bin/date binary for many transformations:
$ date --date="2009-01-02 03:04:05" "+%d %B of %Y at %H:%M and %S seconds"
02 January of 2009 at 03:04 and 05 seconds
This parses the given date and displays it in the chosen format. You can adapt that at will to your needs.
I had a different input time format, so here is a more flexible solution.
Convert dates in BSD/macOS
date -jf in_format [+out_format] in_date
where the formats use strftime (see man strftime).
For the given input format YYYY-MM-DD hh:mm:ss:
$ date -jf '%Y-%m-%d %H:%M:%S' '2017-05-10 13:40:01'
Wed May 10 13:40:01 PDT 2017
To read them into separate variables, I'm taking NVRAM's idea, but allowing you to use any strftime format:
$ date_in='2017-05-10 13:40:01'
$ format='%Y-%m-%d %H:%M:%S'
$ read -r y m d H M S <<< "$(date -jf "$format" '+%Y %m %d %H %M %S' "$date_in")"
$ for var in y m d H M S; do echo "$var=${!var}"; done
y=2017
m=05
d=10
H=13
M=40
S=01
In scripts, always use read -r.
In my case, I wanted to convert between timezones (see your /usr/share/zoneinfo directory for zone names):
$ format=%Y-%m-%dT%H:%M:%S%z
$ TZ=UTC date -jf $format +$format 2017-05-10T02:40:01+0200
2017-05-10T00:40:01+0000
$ TZ=America/Los_Angeles date -jf $format +$format 2017-05-10T02:40:01+0200
2017-05-09T17:40:01-0700
Convert dates in GNU/Linux
On a Mac, you can install the GNU version of date as gdate with brew install coreutils.
date [+out_format] -d in_date
where the out_format uses strftime (see man strftime).
In GNU coreutils' date command, there is no way to explicitly set an input format, since it tries to figure out the input format by itself, and stuff usually just works. (For detail, you can read the manual at coreutils: Date input formats.)
For example:
$ date '+%Y %m %d %H %M %S' -d '2017-05-10 13:40:01'
2017 05 10 13 40 01
To read them into separate variables:
$ read -r y m d H M S <<< "$(date '+%Y %m %d %H %M %S' -d "$date_in")"
To convert between timezones (see your /usr/share/zoneinfo directory for zone names), you can specify TZ="America/Los_Angeles" right in your input string. Note the literal " chars around the zone name, and the space character before in_date:
TZ=out_tz date [+out_format] 'TZ="in_tz" in_date'
For example:
$ format='%Y-%m-%d %H:%M:%S%z'
$ TZ=America/Los_Angeles date +"$format" -d 'TZ="UTC" 2017-05-10 02:40:01'
2017-05-09 19:40:01-0700
$ TZ=UTC date +"$format" -d 'TZ="America/Los_Angeles" 2017-05-09 19:40:01'
2017-05-10 02:40:01+0000
GNU date also understands hour offsets for the time zone:
$ TZ=UTC date +"$format" -d '2017-05-09 19:40:01-0700'
2017-05-10 02:40:01+0000
This is simple, just convert your dashes and colons to a space (no need to change IFS) and use 'read' all on one line:
read Y M D h m s <<< ${date//[-:]/ }
For example:
$ date=$(date +'%Y-%m-%d %H:%M:%S')
$ read Y M D h m s <<< ${date//[-: ]/ }
$ echo "Y=$Y, m=$m"
Y=2009, m=57
$ t='2009-12-03 12:38:15'
$ a=(`echo $t | sed -e 's/[:-]/ /g'`)
$ echo ${a[*]}
2009 12 03 12 38 15
$ echo ${a[3]}
12
The array method is perhaps better, but this is what you were specifically asking for:
IFS=" :-"
read year month day hour minute second < <(echo "YYYY-MM-DD hh:mm:ss")
Pure Bash:
date="2009-12-03 15:35:11"
saveIFS="$IFS"
IFS="- :"
date=($date)
IFS="$saveIFS"
for field in "${date[#]}"
do
echo $field
done
2009
12
03
15
35
11
instead of using the shell scripting,incorporate in your scripting itself like below wheever you need:
a=date +%Y
b=date +%S
c=date +%H
a will be year
b will be seconds
c will be hours. and so on.
Another solution to the OP's problem:
IFS=' -:' read y m d h m s<<<'2014-03-26 16:36:41'
Converting a date to another format with BSD date and GNU date:
$ LC_ALL=C date -jf '%a %b %e %H:%M:%S %Z %Y' 'Wed Mar 26 16:36:41 EET 2014' +%F\ %T
2014-03-26 16:36:41
$ gdate -d 'Wed Mar 26 16:36:41 EET 2014' +%F\ %T
2014-03-26 16:36:41
GNU date recognizes Wed and Mar even in non-English locales but BSD date doesn't.
Converting seconds since epoch to a date and time with GNU date and BSD date:
$ gdate -d #1234567890 '+%F %T'
2009-02-14 01:31:30
$ date -r 1234567890 '+%F %T'
2009-02-14 01:31:30
Converting seconds to hours, minutes, and seconds with a POSIX shell, POSIX awk, GNU date, and BSD date:
$ s=12345;printf '%02d:%02d:%02d\n' $((s/3600)) $((s%3600/60)) $((s%60))
05:25:45
$ echo 12345|awk '{printf "%02d:%02d:%02d\n",$0/3600,$0%3600/60,$0%60}'
05:25:45
$ gdate -d #12345 +%T
05:25:45
$ date -r 12345 +%T
05:25:45
Converting seconds to days, hours, minutes, and seconds:
$ t=12345678
$ printf '%d:%02d:%02d:%02d\n' $((t/86400)) $((t/3600%24)) $((t/60%60)) $((t%60))
142:21:21:18
another pure bash
$ d="2009-12-03 15:35:11"
$ d=${d//[- :]/|}
$ IFS="|"
$ set -- $d
$ echo $1
2009
$ echo $2
12
$ echo $#
2009 12 03 15 35 11
have you tried using cut?
something like this:
dayofweek=date|cut -d" " -f1