Datetime to epoch conversion - bash

Question

I have a bash question (when using awk). I'm extracting every single instance of the first and fifth column in a textfile and piping it to a new file with the following code,
cut -f4 test170201.rawtxt | awk '/stream_0/ { print $1, $5 }' > testLogFile.txt
This is part of the file (test170201.rawtxt) I'm extracting the data from, columns Timestamp and Loss,
Timestamp Stream Status Seq Loss Bytes Delay
17/02/01.10:58:25.212577 stream_0 OK 80281 0 1000 38473
17/02/01.10:58:25.213401 stream_0 OK 80282 0 1000 38472
17/02/01.10:58:25.215560 stream_0 OK 80283 0 1000 38473
17/02/01.10:58:25.216645 stream_0 OK 80284 0 1000 38472
This is the result I'm getting in testLogFile.txt
17/02/01.10:58:25.212577 0
17/02/01.10:58:25.213401 0
17/02/01.10:58:25.215560 0
17/02/01.10:58:25.216645 0
However, I want the Timestamp to be written in epoch in the file above. Is there an easy way of modifying the code I already have to do this?

Answer 1

Given:
$ cat file
Timestamp Stream Status Seq Loss Bytes Delay
17/02/01.10:58:25.212577 stream_0 OK 80281 0 1000 38473
17/02/01.10:58:25.213401 stream_0 OK 80282 0 1000 38472
17/02/01.10:58:25.215560 stream_0 OK 80283 0 1000 38473
17/02/01.10:58:25.216645 stream_0 OK 80284 0 1000 38472
You can write a POSIX Bash script to do what you are looking for:
while IFS= read -r line || [[ -n "$line" ]]; do
if [[ "$line" =~ ^[[:digit:]]{2}/[[:digit:]]{2}/[[:digit:]]{2} ]]
then
arr=($line)
ts=${arr[0]}
dec=${ts##*.} # fractional seconds
# GNU date may need different flags:
epoch=$(date -j -f "%y/%m/%d.%H:%M:%S" "${ts%.*}" "+%s")
printf "%s.%s\t%s\n" "$epoch" "$dec" "${arr[4]}"
fi
done <file >out_file
$ cat out_file
1485975505.212577 0
1485975505.213401 0
1485975505.215560 0
1485975505.216645 0
For GNU date, try:
while IFS= read -r line || [[ -n "$line" ]]; do
if [[ "$line" =~ ^[[:digit:]]{2}/[[:digit:]]{2}/[[:digit:]]{2} ]]
then
arr=($line)
ts="20${arr[0]}"
d="${ts%%.*}"
tmp="${ts%.*}"
tm="${tmp#*.}"
dec="${ts##*.}" # fractional seconds
epoch=$(date +"%s" --date="$d $tm" )
printf "%s.%s\t%s\n" "$epoch" "$dec" "${arr[4]}"
fi
done <file >out_file
For an GNU awk solution, you can do:
awk 'function epoch(s){
split(s, dt, /[/:. ]/)
s="20" dt[1] " " dt[2] " " dt[3] " " dt[4] " " dt[5] " " dt[6]
return mktime(s) "." dt[7]}
/^[0-9][0-9]/ { print epoch($1), $5 }' file >out_file
If you don't want the fractional second included in the epoch, they are easily removed.

Answer 2

awk -F '[.[:blank:]]+' '
# use separator for dot and space (to avoid trailing time info)
{
# for line other than header
if( NR>1) {
# time is set for format "YYYY MM DD HH MM SS [DST]"
# prepare with valuable info
T = "20"$1 " " $2
# use correct separator
gsub( /[\/:]/, " ", T)
# convert to epoch
E = mktime( T)
# print result, adding fractionnal as mentionned later
printf("%d.%d %s\n", E, $3, $7)
}
else {
# print header (line 1)
print $1 " "$7
}
}
' test170201.rawtxt \
> Redirected.file
self commented, code is longer for understanding purpose
use of gnu awk for the mktime function not available in posix or older version
Oneliner a bit optimized here after
awk -F '[.[:blank:]]+' '{if(NR>1){T="20"$1" "$2;gsub(/[\/:]/," ", T);$1=mktime(T)}print $1" "$7}' test170201.rawtxt

Answer 3

Using GNU awk
Input
$ cat f
Timestamp Stream Status Seq Loss Bytes Delay
17/02/01.10:58:25.212577 stream_0 OK 80281 0 1000 38473
17/02/01.10:58:25.213401 stream_0 OK 80282 0 1000 38472
17/02/01.10:58:25.215560 stream_0 OK 80283 0 1000 38473
17/02/01.10:58:25.216645 stream_0 OK 80284 0 1000 38472
Output
$ awk '
BEGIN{cyear = strftime("%y",systime())}
function epoch(v, datetime){
sub(/\./," ",v);
split(v,datetime,/[/: ]/);
datetime[1] = datetime[1] <= cyear ? 2000+datetime[1] : 1900+datetime[1];
return mktime(datetime[1] " " datetime[2] " " datetime[3] " " datetime[4]" " datetime[5]" " datetime[6])
}
/stream_0/{
print epoch($1),$5
}' f
1485926905 0
1485926905 0
1485926905 0
1485926905 0
To write to new file just redirect like below
cut -f4 test170201.rawtxt | awk '
BEGIN{cyear = strftime("%y",systime());}
function epoch(v, datetime){
sub(/\./," ",v);
split(v,datetime,/[/: ]/);
datetime[1] = datetime[1] <= cyear ? 2000+datetime[1] : 1900+datetime[1];
return mktime(datetime[1] " " datetime[2] " " datetime[3] " " datetime[4]" " datetime[5]" " datetime[6])
}
/stream_0/{
print epoch($1),$5
}' > testLogFile.txt