In nginx access.log we have time format in GMT timezone. using awk command,converting logs to csv file.
awk '{print $4","$7,"$10","$15","$16","$17}' </usr/local/nginx/logs/access.log>access.csv
$4 - displays date in GMT (31/Jul/2015:16:03:07).
Kindly guide how to change it to IST and update in that csv file.
If you have access to GNU awk and GNU date, you could use TZ to easily get time at IST:
$ TZ='Asia/Kolkata' date -d "#1438374787"
Where the '1438374787' value is the seconds since epoch in GMT, also known as systime.
To make a systime out of a date, we could use mktime from (GNU) awk:
Convert the GMT string to epoch time directly in awk:
$ echo "a test date 31/07/2015:16:33:07" | awk '{gsub("[/:]"," ",$4); $0=$0;t=sprintf("%d %d %d %d %d %d 0",$6,$5,$4,$7,$8,$9);print(mktime(t))}'
For that to work, you need all values to be numbers (no jul, sorry). If that is not possible, you need to work out a format for an string that the command date could read correctly (sometimes not an easy task).
Placing all in one script:
#!/bin/bash --
echo "a test date 31/07/2015:16:33:07" | awk '{
split($4,A,"[:/]");
gmtdate=sprintf("%d %d %d %d %d %d",A[3],A[2],A[1],A[4],A[5],A[6]);
print gmtdate;
T1=mktime(gmtdate);
print T1;
pt = sprintf ("TZ=%s%s%s date -d %s%s%s%s","\047","Asia/Kolkata","\047","\042","#",T1,"\042");
print pt;
system(pt);
}'
And, running it:
$ ./stackoverflow-awk-time-test.sh
2015 7 31 16 33 7
1438374787
TZ='Asia/Kolkata' date -d "#1438374787"
Sat Aug 1 02:03:07 IST 2015
Several additional debug values are printed from awk (easy to remove).
Related
I have a text log file, the format is like the following
Thread-28689296: Thu Aug 25 15:18:41 2016 [ info ]: xxxxx xxxxxx xxxxx
So I want to run cron job to find some certain error messages in last a few minutes. I wrote the following command
awk -vDate=`date +%b %d %H:%M:%S %Y` -vDate2=`date --date="2 minutes ago" +%b %d %H:%M:%S %Y` '$5 > Date && $5 < Date2' /var/log/dummy.log | grep "Fatal"
In the above command, i search for messages that have a timestamp beween time now and 2 minutes ago with a string Fatal.
But I got the following error
date: extra operand %d'
Try date --help' for more information.
date: extra operand %d'
Try date --help' for more information.
If I run date commands, I got the results as the following
date "+%b %d %H:%M:%S %Y"
Aug 25 15:25:01 2016
date --date="2 minutes ago" +"%b %d %H:%M:%S %Y"
Aug 25 15:31:42 2016
So the date commands in my awk script should be okay.
I also want to redirect the found error messages happening 2 minutes to a file to mail as alert but I did not get that far yet.
Please kindly advice me what is wrong in my awk script. Thanks a lot in advance!
The problem here is with date itself. Let's see how.
You are saying:
vDate2=`date --date="2 minutes ago" +%b %d %H:%M:%S %Y`
Because you want to use
date --date="2 minutes ago" +%b %d %H:%M:%S %Y
However, if you try to run it you'll see that you get the error:
date: extra operand ‘%d’
Try 'date --help' for more information.
The problem is that you need to enclose the FORMAT controls within double quotes:
# v v
$ date --date="2 minutes ago" "+%b %d %H:%M:%S %Y"
Aug 25 14:49:31 2016
When this is done, all together your full awk one-liner can be:
awk -v Date="$(date "+%b %d %H:%M:%S %Y")" \
-v Date2="$(date --date="2 minutes ago" "+%b %d %H:%M:%S %Y")" \
'$5 > Date && $5 < Date2' file
Note I am using -v Date="$(date ...)":
$( ) for process substitution, since backticks ` are almost deprecated, ir at least considered legacy.
date=" things " to prevent errors if the content has spaces.
v var=value using spaces after -v, since -vvar=value is gawk-specific.
I have 2 variables.
GMDCOMTM which stores the date time Tue Oct 1 13:32:40 2013
GMDRRSTM which stores the date time Tue Oct 2 23:35:33 2013
How do I calculate the difference between the 2 variables in hh:mm:ss format and store
it in 3rd variable.? I dont want to use AWK, SED or PERL. I want to use simple shell
script to do it.
Convert dates to %s ---> seconds since 1970-01-01 00:00:00 UTC.
$ date -d"Tue Oct 2 23:35:33 2013" "+%s"
1380749733
So the thing is to get the difference in seconds between both dates using bc as calculator:
$ d1="Tue Oct 1 13:32:40 2013"
$ d2="Tue Oct 2 23:35:33 2013"
$ echo $(date -d"$d2" "+%s") - $(date -d"$d1" "+%s") | bc
122573
Then you can get it into hours, days, with the great function Stéphane Gimenez indicates in UNIX & Linux:
$ displaytime 122573
1 days 10 hours 2 minutes and 53 seconds
C=$(date -d "Tue Oct 1 13:32:40 2013" +%s)
R=$(date -d "Tue Oct 2 23:35:33 2013" +%s)
T=$(date --date=#$((R-C)) +%H:%M:%S)
I'm normal using a custom made function to do the calculations. It may be long way but it will definitely work on all UNIX and Linux based systems. Following the code block.
time_diff(){
foodate1=$1
foodate2=$2
foosecvall=`echo $foodate1 | awk -F: '{ print ($1 * 3600) + ($2 * 60) + $3 }'`
foosecval2=`echo $foodate2 | awk -F: '{ print ($1 * 3600) + ($2 * 60) + $3 }'`
foodiffsec=$((foosecvall-foosecval2));
s=$foodiffsec
h=$((s/3600));
s=$((s-$((h*3600))));
m=$((s/60));
s=$((s-$((m*60))));
footmstm=$h":"$m":"$s
}
Place the above code in your script and then call the function like follows.
TIME1="10:12:14"
TIME2="12:15:14"
time_diff $TIME2 $TIME1
echo $footmstm
I'm sure this answer is obvious but I'm banging my head on it and getting a headache and my Search Foo is failing me…
I have a log file with this date format:
Sep 1 16:55:00 stuff happening
Sep 1 16:55:01 THIS IS THE LINE YOU WANT at this time stamp
Sep 1 16:55:02 more stuff
Sep 1 16:55:02 THIS IS THE LINE YOU WANT at this time stamp
Sep 1 16:55:03 blah
Sep 1 16:55:04 blah and so on…..
My ultimate goal is to:
Find the last line in the log file with a given string eg: "THIS IS THE LINE…" this is my "magic time" that I will do calculations on later.
Take the date of that line and set a variable that is the date +NN seconds. The time in the future will usually just short of 24hrs in the future from the time in step 1 so crossing into the next day may happen if that is important.
At some point in the script, advance the system clock to the new date/time after which I will be checking for certain events to fire.
I know this is way wrong but so far I have figured out how to:
Grab the last date stamp for my event.
logDate=cat /logdir/my.log | grep "THIS IS THE LINE" | tail -1 | cut -f1,2,3 -d" "
Returns: Sept 1 16:55:02
Convert the date into a more usable format
logDate2="$(date -d "$logDate" +"%m-%d %H:%M:%S")"; echo $logDate2
Returns: 09-17 16:55:02
I'm stuck here - what I want is:
futuredate=$logdate2 + XXXSeconds
Could someone help me with the time calculation or perhaps point out a better way to do all of this?
Thanks.
I'm stuck here - what I want is:
futuredate=$logdate2 + XXXSeconds
You can do it by converting through timestamps:
# convert log date to timestamp
logts="$(date -d "$logDate" '+%s')"
# add timestamp with seconds
futurets=$(( logts + XXXSeconds ))
# get date based from timestamp, optionally you can add a format.
futuredate=$(date -d "#${futurets}")
# Get time in seconds from the epoc (1970-01-01 00:00:00 UTC)
dateinseconds=$(date +"%s" -d "$(tail -1 logfile | grep "THIS IS THE LINE" | awk '{print $1, $2, $3}')")
# You can also use just awk without grep and tail to match and print the last line
dateinseconds=$(date +"%s" -d "$(awk '{/THIS IS THE LINE/}END{print $1, $2, $3}' logfile)")
gotofuture=$(( $dateinseconds + 2345 )) # Add 2345 seconds
newdate=$(date -d "#${gotofuture}")
echo "$newdate"
I'm trying to work on a logfile, and I need to be able to specify the range of dates. So far (before any processing), I'm converting a date/time string to timestamp using date --date "monday" +%s.
Now, I want to be able to iterate over each line in a file, but check if the date (in a human readable format) is within the allowed range. To do this, I'd like to do something like the following:
echo `awk '{if(`date --date "$3 $4 $5 $6 $7" +%s` > $START && `date --date "" +%s` <= $END){/*processing code here*/}}' myfile`
I don't even know if thats possible... I've tried a lot of variations, plus I couldn't find anything understandable/usable online.
Thanks
Update:
Example of myfile is as follows. Its logging IPs and access times:
123.80.114.20 Sun May 01 11:52:28 GMT 2011
144.124.67.139 Sun May 01 16:11:31 GMT 2011
178.221.138.12 Mon May 02 08:59:23 GMT 2011
Given what you have to do, its really not that hard AND it is much more efficient to do your date processing by converting to strings and comparing.
Here's a partial solution that uses associative arrays to convert the month value to a number. Then you rely on the %02d format specifier to ensure 2 digits. You can reformat the dateTime value with '.', etc or leave the colons in the hr:min:sec if you really need the human readability.
The YYYYMMDD format is a big help in these sort of problems, as LT, GT, EQ all work without any further formatting.
echo "178.221.138.12 Mon May 02 08:59:23 GMT 2011" \
| awk 'BEGIN {
mons["Jan"]=1 ; mons["Feb"]=2; mons["Mar"]=3
mons["Apr"]=4 ; mons["May"]=5; mons["Jun"]=6
mons["Jul"]=7 ; mons["Aug"]=8; mons["Sep"]=9
mons["Oct"]=10 ; mons["Nov"]=11; mons["Dec"]=12
}
{
# 178.221.138.12 Mon May 02 08:59:23 GMT 2011
printf("dateTime=%04d%02d%02d%02d%02d%02d\n",
$NF, mons[$3], $4, substr($5,1,2), substr($5,4,2), substr($5,7,2) )
} ' -v StartTime=20110105235959
The -v StartTime is ilustrative of how to pass in (and the matching format) your starTime value.
I hope this helps.
Here's an alternative approach using awk's built-in mktime() function. I've never bothered with the month parsing until now - thanks to shelter for that part (see accepted answer). It always feels time to switch language around that point.
#!/bin/bash
# input format:
#(1 2 3 4 5 6 7)
#123.80.114.20 Sun May 01 11:52:28 GMT 2011
awk -v startTime=1304252691 -v endTime=1306000000 '
BEGIN {
mons["Jan"]=1 ; mons["Feb"]=2; mons["Mar"]=3
mons["Apr"]=4 ; mons["May"]=5; mons["Jun"]=6
mons["Jul"]=7 ; mons["Aug"]=8; mons["Sep"]=9
mons["Oct"]=10 ; mons["Nov"]=11; mons["Dec"]=12;
}
{
hmsSpaced=$5; gsub(":"," ",hmsSpaced);
timeInSec=mktime($7" "mons[$3]" "$4" "hmsSpaced);
if (timeInSec > startTime && timeInSec <= endTime) print $0
}' myfile
(I've chosen example time thresholds to select only the last two log lines.)
Note that if the mktime() function were a bit smarter this whole thing would reduce to:
awk -v startTime=1304252691 -v endTime=1306000000 't=mktime($7" "$3" "$4" "$5); if (t > startTime && t <= endTime) print $0}' myfile
I'm not sure of the format of the data you're parsing, but I do know that you can't use the backticks within single quotes. You'll have to use double quotes. If there are too many quotes being nested, and it's confusing you, you can also just save the output of your date command to a variable beforehand.
How would you parse a date in bash, with separate fields (years, months, days, hours, minutes, seconds) into different variables?
The date format is: YYYY-MM-DD hh:mm:ss
Does it have to be bash? You can use the GNU coreutils /bin/date binary for many transformations:
$ date --date="2009-01-02 03:04:05" "+%d %B of %Y at %H:%M and %S seconds"
02 January of 2009 at 03:04 and 05 seconds
This parses the given date and displays it in the chosen format. You can adapt that at will to your needs.
I had a different input time format, so here is a more flexible solution.
Convert dates in BSD/macOS
date -jf in_format [+out_format] in_date
where the formats use strftime (see man strftime).
For the given input format YYYY-MM-DD hh:mm:ss:
$ date -jf '%Y-%m-%d %H:%M:%S' '2017-05-10 13:40:01'
Wed May 10 13:40:01 PDT 2017
To read them into separate variables, I'm taking NVRAM's idea, but allowing you to use any strftime format:
$ date_in='2017-05-10 13:40:01'
$ format='%Y-%m-%d %H:%M:%S'
$ read -r y m d H M S <<< "$(date -jf "$format" '+%Y %m %d %H %M %S' "$date_in")"
$ for var in y m d H M S; do echo "$var=${!var}"; done
y=2017
m=05
d=10
H=13
M=40
S=01
In scripts, always use read -r.
In my case, I wanted to convert between timezones (see your /usr/share/zoneinfo directory for zone names):
$ format=%Y-%m-%dT%H:%M:%S%z
$ TZ=UTC date -jf $format +$format 2017-05-10T02:40:01+0200
2017-05-10T00:40:01+0000
$ TZ=America/Los_Angeles date -jf $format +$format 2017-05-10T02:40:01+0200
2017-05-09T17:40:01-0700
Convert dates in GNU/Linux
On a Mac, you can install the GNU version of date as gdate with brew install coreutils.
date [+out_format] -d in_date
where the out_format uses strftime (see man strftime).
In GNU coreutils' date command, there is no way to explicitly set an input format, since it tries to figure out the input format by itself, and stuff usually just works. (For detail, you can read the manual at coreutils: Date input formats.)
For example:
$ date '+%Y %m %d %H %M %S' -d '2017-05-10 13:40:01'
2017 05 10 13 40 01
To read them into separate variables:
$ read -r y m d H M S <<< "$(date '+%Y %m %d %H %M %S' -d "$date_in")"
To convert between timezones (see your /usr/share/zoneinfo directory for zone names), you can specify TZ="America/Los_Angeles" right in your input string. Note the literal " chars around the zone name, and the space character before in_date:
TZ=out_tz date [+out_format] 'TZ="in_tz" in_date'
For example:
$ format='%Y-%m-%d %H:%M:%S%z'
$ TZ=America/Los_Angeles date +"$format" -d 'TZ="UTC" 2017-05-10 02:40:01'
2017-05-09 19:40:01-0700
$ TZ=UTC date +"$format" -d 'TZ="America/Los_Angeles" 2017-05-09 19:40:01'
2017-05-10 02:40:01+0000
GNU date also understands hour offsets for the time zone:
$ TZ=UTC date +"$format" -d '2017-05-09 19:40:01-0700'
2017-05-10 02:40:01+0000
This is simple, just convert your dashes and colons to a space (no need to change IFS) and use 'read' all on one line:
read Y M D h m s <<< ${date//[-:]/ }
For example:
$ date=$(date +'%Y-%m-%d %H:%M:%S')
$ read Y M D h m s <<< ${date//[-: ]/ }
$ echo "Y=$Y, m=$m"
Y=2009, m=57
$ t='2009-12-03 12:38:15'
$ a=(`echo $t | sed -e 's/[:-]/ /g'`)
$ echo ${a[*]}
2009 12 03 12 38 15
$ echo ${a[3]}
12
The array method is perhaps better, but this is what you were specifically asking for:
IFS=" :-"
read year month day hour minute second < <(echo "YYYY-MM-DD hh:mm:ss")
Pure Bash:
date="2009-12-03 15:35:11"
saveIFS="$IFS"
IFS="- :"
date=($date)
IFS="$saveIFS"
for field in "${date[#]}"
do
echo $field
done
2009
12
03
15
35
11
instead of using the shell scripting,incorporate in your scripting itself like below wheever you need:
a=date +%Y
b=date +%S
c=date +%H
a will be year
b will be seconds
c will be hours. and so on.
Another solution to the OP's problem:
IFS=' -:' read y m d h m s<<<'2014-03-26 16:36:41'
Converting a date to another format with BSD date and GNU date:
$ LC_ALL=C date -jf '%a %b %e %H:%M:%S %Z %Y' 'Wed Mar 26 16:36:41 EET 2014' +%F\ %T
2014-03-26 16:36:41
$ gdate -d 'Wed Mar 26 16:36:41 EET 2014' +%F\ %T
2014-03-26 16:36:41
GNU date recognizes Wed and Mar even in non-English locales but BSD date doesn't.
Converting seconds since epoch to a date and time with GNU date and BSD date:
$ gdate -d #1234567890 '+%F %T'
2009-02-14 01:31:30
$ date -r 1234567890 '+%F %T'
2009-02-14 01:31:30
Converting seconds to hours, minutes, and seconds with a POSIX shell, POSIX awk, GNU date, and BSD date:
$ s=12345;printf '%02d:%02d:%02d\n' $((s/3600)) $((s%3600/60)) $((s%60))
05:25:45
$ echo 12345|awk '{printf "%02d:%02d:%02d\n",$0/3600,$0%3600/60,$0%60}'
05:25:45
$ gdate -d #12345 +%T
05:25:45
$ date -r 12345 +%T
05:25:45
Converting seconds to days, hours, minutes, and seconds:
$ t=12345678
$ printf '%d:%02d:%02d:%02d\n' $((t/86400)) $((t/3600%24)) $((t/60%60)) $((t%60))
142:21:21:18
another pure bash
$ d="2009-12-03 15:35:11"
$ d=${d//[- :]/|}
$ IFS="|"
$ set -- $d
$ echo $1
2009
$ echo $2
12
$ echo $#
2009 12 03 15 35 11
have you tried using cut?
something like this:
dayofweek=date|cut -d" " -f1