I am trying to figure out how to parse a file with ISO 8601-formatted time stamps, add one second and then output them to a file.
All the examples I have found don't really tell me how to do it with ISO 8601 date/time strings.
Example:
read a csv of times like: "2017-02-15T18:47:59" (some are correct, others are not)
and spit out in a new file "2017-02-15T18:48:00"
mainly just trying to correct a bunch of dates that have 59 seconds at the end to round up to the 1 second mark.
This is my current progress:
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
# startd=$(date -j -f '%Y%m%d' "$line" +'%Y%m%d');
# echo "$startd";
startd=$(date -j -u -f "%a %b %d %T %Z %Y" $line)
#startd=$(date -j -f '%Y%m%d' "$line" +'%Y%m%d');
echo "$startd";
done < "$1"
Any help would be appreciated
jm666's helpful perl answer will be much faster than your shell loop-based approach.
That said, if you want to make your bash code work on macOS, with its BSD date implementation, here's a solution:
# Define the input date format, which is also used for output.
fmt='%Y-%m-%dT%H:%M:%S'
# Note: -j in all date calls below is needed to suppress setting the
# system date.
while IFS= read -r line || [[ -n "$line" ]]; do
# Parse the line at hand using input format specifier (-f) $fmt,
# and output-format specifier (+) '%s', which outputs a Unix epoch
# timestamp (in seconds).
ts=$(date -j -f "$fmt" "$line" +%s)
# See if the seconds-component (%S) is 59...
if [[ $(date -j -f %s "$ts" +%S) == '59' ]]; then
# ... and, if so, add 1 second (-v +1S).
line=$(date -j -f %s -v +1S "$ts" +"$fmt")
fi
# Output the possibly adjusted timestamp.
echo "$line"
done < "$1"
Note that input dates such as 2017-02-15T18:47:59 are interpreted as local time, because they contain no time-zone information.
This could do the job
perl -MTime::Piece -nlE '$f=q{%Y-%m-%dT%H:%M:%S};$t=Time::Piece->strptime($_,$f)+1;say $t->strftime($f)' < dates.txt
if the dates.txt contains
2017-02-15T18:47:59
2016-02-29T23:59:59
2017-02-28T23:59:59
2015-12-31T23:59:59
2010-10-10T10:10:10
the above produces
2017-02-15T18:48:00
2016-03-01T00:00:00
2017-03-01T00:00:00
2016-01-01T00:00:00
2010-10-10T10:10:11
Related
I have this script (which works fine) that will write all the date/time per second, from a start date/time till an end date/time to a file
while read line; do
FIRST_TIMESTAMP="20230109-05:00:01" #this is normally a variable that changes with each $line
LAST_TIMESTAMP="20230112-07:00:00" #this is normally a variable that changes with each $line
date=$FIRST_TIMESTAMP
while [[ $date < $LAST_TIMESTAMP || $date == $LAST_TIMESTAMP ]]; do
date2=$(echo $date |sed 's/ /-/g' |sed "s/^/'/g" |sed "s/$/', /g")
echo "$date2" >> "OUTPUTFOLDER/output_LABELS_$line"
date=$(date -d "$date +1 sec" +"%Y%m%d %H:%M:%S")
done
done < external_file
However this sometimes needs to run 10 times, and the start date/time and end date/time sometimes lies days apart.
Which makes the script take a long time to write all that data.
Now I am wondering if there is a faster way to do this.
Avoid using a separate date call for each date. In the next example I added a safety parameter maxloop, avoiding loosing resources when the dates are wrong.
#!/bin/bash
awkdates() {
maxloop=1000000
awk \
-v startdate="${first_timestamp:0:4} ${first_timestamp:4:2} ${first_timestamp:6:2} ${first_timestamp:9:2} ${first_timestamp:12:2} ${first_timestamp:15:2}" \
-v enddate="${last_timestamp:0:4} ${last_timestamp:4:2} ${last_timestamp:6:2} ${last_timestamp:9:2} ${last_timestamp:12:2} ${last_timestamp:15:2}" \
-v maxloop="${maxloop}" \
'BEGIN {
T1=mktime(startdate);
T2=mktime(enddate);
linenr=1;
while (T1 <= T2) {
printf("%s\n", strftime("%Y%m%d %H:%M:%S",T1));
T1+=1;
if (linenr++ > maxloop) break;
}
}'
}
mkdir -p OUTPUTFOLDER
while IFS= read -r line; do
first_timestamp="20230109-05:00:01" #this is normally a variable that changes with each $line
last_timestamp="20230112-07:00:00" #this is normally a variable that changes with each $line
awkdates >> "OUTPUTFOLDER/output_LABELS_$line"
done < <(printf "%s\n" "line1" "line2")
Using epoch time (+%s and #) with GNU date and GNU seq to
produce datetimes in ISO 8601 date format:
begin=$(date -ud '2023-01-12T00:00:00' +%s)
end=$(date -ud '2023-01-12T00:00:12' +%s)
seq -f "#%.0f" "$begin" 1 "$end" |
date -uf - -Isec
2023-01-12T00:00:00+00:00
2023-01-12T00:00:01+00:00
2023-01-12T00:00:02+00:00
2023-01-12T00:00:03+00:00
2023-01-12T00:00:04+00:00
2023-01-12T00:00:05+00:00
2023-01-12T00:00:06+00:00
2023-01-12T00:00:07+00:00
2023-01-12T00:00:08+00:00
2023-01-12T00:00:09+00:00
2023-01-12T00:00:10+00:00
2023-01-12T00:00:11+00:00
2023-01-12T00:00:12+00:00
if you're using macOS/BSD's date utility instead of the gnu one, the equivalent command to parse would be :
(bsd)date -uj -f '%FT%T' '2023-01-12T23:34:45' +%s
1673566485
...and the reverse process is using -r flag instead of -d, sans "#" prefix :
(bsd)date -uj -r '1673566485' -Iseconds
2023-01-12T23:34:45+00:00
(gnu)date -u -d '#1673566485' -Iseconds
2023-01-12T23:34:45+00:00
I'm creating a list of directories with the requested date range in their name.
Directories are all labeled other_2019-07-18T00-00-00. The T is messing me up!
Copied this loop from somewhere.
#!/bin/bash
curdate=$(date +%Y-%m-%d%H-%M-%S)
#
for o in other_*; do
tmp=${o##other_}
tmp=$(echo "$tmp" | sed 's/T//') # clean up prefixes
fdate=$(date -d "${tmp}")
(( curdate < fdate )) && echo "$o"
done
I expect the final echo to include the path of all dir that match.
Unlike AWK, Bash comparison operator < works only on numerical values.
Please try instead:
#!/bin/bash
curdate=$(date +%Y%m%d%H%M%S)
for o in other_*; do
tmp=${o##other_}
fdate=$(echo "$tmp" | sed 's/[-T]//g') # numeralization
(( curdate < fdate )) && echo "$o"
done
As an alternative, you can compare epoch times:
#!/bin/bash
curdate=$(date +%s)
for o in other_*; do
tmp=${o##other_}
tmp=$(echo "$tmp" | sed 's/T/ /' | sed 's/\([0-9][0-9]\)-\([0-9][0-9]\)-\([0-9][0-9]\)$/\1:\2:\3/')
fdate=$(date -d "$tmp" +%s)
(( curdate < fdate )) && echo "$o"
done
Hope this helps.
Instead of dropping T...
date -d 2019-03-23T00-06-28
date: invalid date '2019-03-23T00-06-28'
ok, but:
date -d 2019-03-23T00:06:28
Sat Mar 23 00:06:28 UTC 2019
So we have to replace last two dashes by ::
As your question is tagged bash:
file="somepath/other_2019-07-18T00-00-00.extension"
time=${file#*other_} # suppress from left until 'other_'
time=${time%.*} # suppress extension
time=${time//-/:} # replace all dash by a `:`
time=${time/:/-} # replace 1nd `:` by a dash
time=${time/:/-} # replace 1nd `:` by a dash (again)
date -d $time
Thu Jul 18 00:00:00 UTC 2019
This could by written:
printf -v now "%(%s)T" -1 # bashism for current date to variable $now
for file in somepath/other_*.ext ;do
time=${file#*other_} time=${time%.*} time=${time//-/:}
time=${time/:/-} time=${time/:/-}
read fdate < <(date +%s -d $time)
((fdate > now)) && { echo $file: $((fdate - now)) ; }
done
Reducing forks (to date) improve quickness:
for matching your sample, you could replace for file in somepath/other_*.ext ;do by for file in other_*; do. This must work quite same.
fifo=/tmp/fifo-date-$$
mkfifo $fifo
exec 5> >(exec stdbuf -o0 date -f - +%s >$fifo 2>&1)
echo now 1>&5
exec 6< $fifo
read -t 1 -u 6 now
rm $fifo
for file in otherdates/*.ext ; do
time=${file#*other_} time=${time%.*} time=${time//-/:}
time=${time/:/-} time=${time/:/-}
echo $time 1>&5 && read -t 1 -u 6 fdate
((fdate > now)) && {
echo $file: $((fdate - now))
}
done
exec 6>&-
exec 5>&-
In this, I run date +%s in background, with -f argument, date will interpret each incoming line, then answer UNIX_TIME. So $now is first read from date process, by:
echo now >&5 && # now, the string
read -u 6 now # read will populate `$now` variable
Nota, once fifo opened by both input and output, they could be deleted. It will remain for process until process close them.
there is no space between day and hour, which causes date not to be able to read the date. Try:
sed 's/T/ /'
I have two dates as follows:
2019-01-06 00:02:10 | END
2019-01-05 23:52:00 | START
How could I calculate and print the difference between START and END dates in seconds?
For above case I would like to get something like:
610
Assuming GNU implementation based OS, you can use date's option %s and -d to calculate the time difference in seconds using command substitution and arithmetic operations.
START="2019-01-05 23:52:00"
END="2019-01-06 00:02:10"
Time_diff_in_secs=$(($(date -d "$END" +%s) - $(date -d "$START" +%s)))
echo $Time_diff_in_secs
Output:
610
Hope this helps!!!
With bash and GNU date:
while read d t x x; do
[[ $x == "END" ]] && end="$d $t"
[[ $x == "START" ]] && start="$d $t"
done < file
end=$(date -u -d "$end" '+%s')
start=$(date -u -d "$start" '+%s')
diff=$(($end-$start))
echo "$diff"
Output:
610
See: man date
What you're asking for is difficult verging on impossible using pure bash. Bash doesn't have any date functions of its own. For date processing, most recommendations you'll get will be to use your operating system's date command, but the usage of this command varies by operating system.
In BSD (including macOS):
start="2019-01-05 23:52:00"; end="2019-01-06 00:02:10"
printf '%d\n' $(( $(date -j -f '%F %T' "$end" '+%s') - $(date -j -f '%F %T' "$start" '+%s') ))
In Linux, or anything using GNU date (possibly also Cygwin):
printf '%d\n' $(( $(date -d "$end" '+%s') - $(date -d "$start" '+%s') ))
And just for the fun of it, if you can't (or would prefer not to) use date for some reason, you might be able to get away with gawk:
gawk 'END{ print mktime(gensub(/[^0-9]/," ","g",end)) - mktime(gensub(/[^0-9]/," ","g",start)) }' start="$start" end="$end" /dev/null
The mktime() option parses a date string in almost exactly the format you're providing, making the math easy.
START="2019-01-05 23:52:00"
END="2019-01-06 00:02:10"
parse () {
local data=(`grep -oP '\d+' <<< "$1"`)
local y=$((${data[0]}*12*30*24*60*60))
local m=$((${data[1]}*30*24*60*60))
local d=$((${data[2]}*24*60*60))
local h=$((${data[3]}*60*60))
local mm=$((${data[4]}*60))
echo $((y+m+d+h+mm+${data[5]}))
}
START=$(parse "$START")
END=$(parse "$END")
echo $((END-START)) // OUTPUT: 610
Was trying to solve the same problem on a non-GNU OS, i.e. macOS. I couldn't apply any of the solutions above, although it inspired me to come up with the following solution. I am using some in-line Ruby from within my shell script, which should work out of the box on macOS.
START="2019-01-05 23:52:00"
END="2019-01-06 00:02:10"
SECONDS=$(ruby << RUBY
require 'date'
puts ((DateTime.parse('${END}') - DateTime.parse('${START}')) * 60 * 60 * 24).to_i
RUBY)
echo ${SECONDS}
# 610
I'm writing a bash script that reads and parses data from log lines, and it's trying to convert the format. The script is long, but I'll focus on the error:
if [[ $line == *"Marked: Malicious, To:"* ]]; then
QUEUE_ID=`awk ' {print $13}' <<< $line | sed 's/,//g'` # QUEUE ID... index of the array
F_TIME["X$QUEUE_ID"]=`awk '{print $3}' <<< $line ` # Hour
F_DAY["X$QUEUE_ID"]=`awk '{print $2}' <<< $line ` # Day
F_MONTH["X$QUEUE_ID"]=`awk '{print $1}' <<< $line ` # Month
#Procesing and obtaining the diff
F_FULLTIME["X$QUEUE_ID"]="${F_DAY["X$QUEUE_ID"]}-${F_MONTH["X$QUEUE_ID"]}-$year-${F_TIME["X$QUEUE_ID"]}" # creating the time in desired format
s=`date -j -f "%d-%b-%Y-%H:%M:%S" "${F_FULLTIME[X$QUEUE_ID]}" +%s` # ERROR LINE!! Doesn't work!
#s=`date -j -f "%d-%b-%Y-%H:%M:%S" "2-Mar-2016-22:24:33" +%s` # Test line
echo ".....${F_FULLTIME["X$QUEUE_ID"]}....2-Mar-2016-22:24:33...."
echo "time $s ...\n";
fi
Test process: I comment the test line (marked on the script), and I try to do conversion to variable s. This is the output for each key:
Failed conversion of ``2-Mar-2016-20:30:03'' using format ``%d-%b-%Y-%H:%M:%S''
date: illegal time format
usage: date [-jnu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
.....2-Mar-2016-20:30:03....2-Mar-2016-22:24:33....
Date is not taking the format. And as you can see, desired format and the format I have on the variable are pretty much same.
Test #2: I comment the error line, and I remove comment the test line. This is the result:
.....2-Mar-2016-22:06:10....2-Mar-2016-22:24:33....
time 1456953873 ...
Works perfectly. And of course text is aparently same as my variable on the first try.
The code looks OK, although overly complex. Try this simplified version, which should be a little easier to debug. We're going to work with individual variables as much as possible, rather than array entries.
if [[ $line == *"Marked: Malicious, To:"* ]]; then
# run `awk` once
fields=$(awk '{print $1, $2, $3, $13}' <<< "$line")
read f_month f_day f_time queue_id <<< "$fields"
queue_id=${queue_id//,/}
# Is the X really necessary?
# F_TIME["X$queue_id"]=$f_time
# F_DAY["X$queue_id"]=$f_day
# F_MONTH["X$queue_id"]=$f_month
#Procesing and obtaining the diff
f_fulltime="$f_day-$f-month-$year-$f_time"
# F_FULLTIME["X$queue_id"]=$f_fulltime
s=$(date -j -f "%d-%b-%Y-%H:%M:%S" "$f_fulltime" )
fi
Some questions to ask:
What are the exact values of queue_id, f_time, f_day, and f_month? Use, e.g., printf '%s' "$queue_id" | hexdump -C to see if
there are any hidden characters from $line that are confusing date.
Can you reproduce the problem by hardcoding the values of the pieces instead of just f_fulltime itself?
Can you reproduce the problem by hard-coding the value of line just prior to the call to awk?
Well.. I've clean the script.., Remove extra variables, declared arrays, and I found where the error comes.. but not sure why. The error comes on the variable $year.
This is defined as:
year=`date +"%Y"`
Then.. when I tried this conversion:
s_fulltime["X$queue_id"]="$day-$month-$year-$time" >> date: illegal time format
But.. if I harcode "2016" then..
year="2016"
All works fine..
Why?
I have date and time format like this(yearmonthday):
20141105 11:30:00
I need assignment year, month, day, hour and minute values to variable.
I can do it year, day and hour like this:
year=$(awk '{print $1}' log.log | sed 's/^\(....\).*/\1/')
day=$(awk '{print $1}' log.log | sed 's/^.*\(..\).*/\1/')
hour=$(awk '{print $2}' log.log | sed 's/^\(..\).*/\1/')
How can I do this for month and minute?
--
And I need that every line of my log file:
20141105 11:30:00 /bla/text.1
20141105 11:35:00 /bla/text.2
20141105 11:40:00 /bla/text.3
....
I'm trying read line by line this log file and do this:
mkdir -p "/bla/backup/$year/$month/$day/$hour/$minute"
mv $file "/bla/backup/$year/$month/$day/$hour/$minute"
Here is my not working code:
#!/bin/bash
LOG=/var/log/LOG
while read line
do
year=${line:0:4}
month=${line:4:2}
day=${line:6:2}
hour=${line:9:2}
minute=${line:12:2}
file=$(awk '{print $3}')
if [ -f "$file" ]; then
printf -v path "%s/%s/%s/%s/%s" $year $month $day $hour $minute
mkdir -p "/bla/backup/$path"
mv $file "/bla/backup/$path"
fi
done < $LOG
You don't need to call out to awk to date at all, use bash's substring operations
d="20141105 11:30:00"
yr=${d:0:4}
mo=${d:4:2}
dy=${d:6:2}
hr=${d:9:2}
mi=${d:12:2}
printf -v dir "/bla/%s/%s/%s/%s/%s/\n" $yr $mo $dy $hr $mi
echo "$dir"
/bla/2014/11/05/11/30/
Or directly, without all the variables.
printf -v dir "/bla/%s/%s/%s/%s/%s/\n" ${d:0:4} ${d:4:2} ${d:6:2} ${d:9:2} ${d:12:2}
Given your log file:
while read -r date time file; do
d="$date $time"
printf -v dir "/bla/%s/%s/%s/%s/%s/\n" ${d:0:4} ${d:4:2} ${d:6:2} ${d:9:2} ${d:12:2}
mkdir -p "$dir"
mv "$file" "$dir"
done < filename
or, making a big assumption that there are no whitespace or globbing characters in your filenames:
sed -r 's#(....)(..)(..) (..):(..):.. (.*)#mv \6 /blah/\1/\2/\3/\4/\5#' | sh
date command also do this work
#!/bin/bash
year=$(date +'%Y' -d'20141105 11:30:00')
day=$(date +'%d' -d'20141105 11:30:00')
month=$(date +'%m' -d'20141105 11:30:00')
minutes=$(date +'%M' -d'20141105 11:30:00')
echo "$year---$day---$month---$minutes"
You can use only one awk
month=$(awk '{print substr($1,5,2)}' log.log)
year=$(awk '{print substr($1,0,4)}' log.log)
minute=$(awk '{print substr($2,4,2)}' log.log)
etc
I guess you are processing the log file, which each line starts with the date string. You may have already written a loop to handle each line, in your loop, you could do:
d="$(awk '{print $1,$2}' <<<"$line")"
year=$(date -d"$d" +%Y)
month=$(date -d"$d" +%m)
day=$(date -d"$d" +%d)
min=$(date -d"$d" +%M)
Don't repeat yourself.
d='20141105 11:30:00'
IFS=' ' read -r year month day min < <(date -d"$d" '+%Y %d %m %M')
echo "year: $year"
echo "month: $month"
echo "day: $day"
echo "min: $min"
The trick is to ask date to output the fields you want, separated by a character (here a space), to put this character in IFS and ask read to do the splitting for you. Like so, you're only executing date once and only spawn one subshell.
If the date comes from the first line of the file log.log, here's how you can assign it to the variable d:
IFS= read -r d < log.log
eval "$(
echo '20141105 11:30:00' \
| sed 'G;s/\(....\)\(..\)\(..\) \(..\):\(..\):\(..\) *\(.\)/Year=\1\7Month=\2\7Day=\3\7Hour=\4\7Min=\5\7Sec=\6/'
)"
pass via a assignation string to evaluate. You could easily adapt to also check the content by replacing dot per more specific pattern like [0-5][0-9] for min and sec, ...
posix version so --posix on GNU sed
I wrote a function that I usually cut and paste into my script files
function getdate()
{
local a
a=(`date "+%Y %m %d %H %M %S" | sed -e 's/ / /'`)
year=${a[0]}
month=${a[1]}
day=${a[2]}
hour=${a[3]}
minute=${a[4]}
sec=${a[5]}
}
in the script file, on a line of it's own
getdate
echo "year=$year,month=$month,day=$day,hour=$hour,minute=$minute,second=$sec"
Of course, you can modify what I provided or use answer [6] above.
The function takes no arguments.