I want to display the duration of a command in my script. Depeding the duration I want somethng like this: 12 minutes 04 seconds or 01 hours 15 minutes 02 seconds. Always with a leading zero.
I tried different things I found here, but didn't get the result.
BTW: that's my first try in the bash.
#!/bin/bash
DATE1=$(date +%s)
# ... your commands
DATE2=$(date +%s)
DIFF=$(awk -vdate1=$DATE1 -vdate2=$DATE2 'BEGIN{print strftime("%H hour %M minutes %S seconds",date2-date1,1)}')
echo $DIFF
Time is converted to seconds and stored in variables DATE1 and DATE2
pre-condition DATE2 > DATE1
DATE1=$(date +%s)
# ... your commands
DATE2=$(date +%s)
strftime is used to get time-diff in seconds and formatted
1 is passed as 3rd argument as UTC Flag
strftime("%H hour %M minutes %S seconds",date2-date1,1)
You could use the time executable (not the bash builtin because the external program features the format option -f and delivers the time as [hours:]minutes:seconds and I am not in the mood right now to wait an hour to find out how the builtin shows us the hours :) ) and awk like this (using the example sleep 2):
/usr/bin/time -f "%E" sleep 2 2> >(
awk -F : '
{ if(FN>2) printf("%02d hours %02d minutes %02d seconds\n", $1, $2, $3)
else printf("%02d minutes %02d seconds\n", $1, $2)
}'
)
Here we use /usr/bin/time with its -f option. Then we pipe the output of stderr into awk splitting the string at :. (time writes to stderr, thus we need the 2> to redirect stderr into the >( awk ... ) filter.
The awk filter decides on the number of fields in NF what printf statement it is using.
Related
my log has this date format at the beginning of each line:
2018 Sep 21 17:16:27:796
I need to grep the last 10 minutes of this log... any help?
my current experiments:
tenminutesago=$(date --date='10 minutes ago' +"%Y %b %e %H:%M:%S"):999
My idea was to convert the log format to a progressive number and then check everything greater than that number.
I see that the command: date +"%Y %b %e %H:%M:%S" gives a date in the same format of the log. The command: date +"%Y%m%e%H%M%S" gives a date in a progressive number (201810041204019)
You could do
for i in {10..0}; do
d=$(date -d "$i minutes ago" +'%Y %b %e %H:%M')
grep "$d" logfile
done
This just divides the problem in the 11 sequential subtasks of getting all lines from 10 minutes ago, all lines from 9 minutes ago, etc. until the current minute.
Edit:
Here's an alternate solution that prints all lines following the first one where a date stamp from the last 10 minutes was found, not only those that carry a date stamp, and also avoids reading the file over from start several times:
# build a regex pattern that matches any date in the given format from the last 10 minutes
pattern=$(date +'%Y %b %e %H:%M')
for i in {10..1}; do
pattern+=\|$(date -d "$i minutes ago" +'%Y %b %e %H:%M')
done
# print all lines starting from the first one that matches one of the dates in the pattern
awk "/$pattern/,0" logfile
Under the assumption that your loglines looks like
YYYY Bbb dd HH:MM:SS:sss Some random log message is here
You can do the following:
awk -v d=$(date -d "10 minutes ago" "+%Y %m %d %T") '
{ mm = sprintf("%0.2d",(index("JanFebMarAprMayJunJulAugSepOctNovDec",$2)+2)/3)
s = $1 " " mm " " $3 " "$4 }
(s >= d){print}' logfile
The idea is to convert your date format into a Sortable format (Note that "Jan" < "Mar" but "Feb" < "Jan"). This is done by converting your month into a number with two digits and then compare it stringwise against the correct date.
Try your current approach without the seconds and milliseconds.
tenminutesago=$(date --date='10 minutes ago' +"%Y %b %e %H:%M")
Is not exactly the last ten minutes to a second level, but I think it is enough for most of the cases. That will give you the first line in the log within the time window. Now you can get the total lines and subtract the line number of your previous grep, and then tail the file. The script could be like this:
LOGFILE="filename.log"
tenminutesago=$(date --date='9000 minutes ago' +"%Y %b %e %H:%M") # matching pattern
tlines=$(cat $LOGFILE | wc -l) # Total lines in file
let lines=$tlines-$(grep -n "$tenminutesago" $LOGFILE | grep -m 1 -oP "^[0-9]*" || echo $tlines) # lines after matching occurence
echo "$lines lines FOUND from the last X minutes"
tail -n $lines $LOGFILE # last lines in file
As suggested by #Gem Taylor, this could be reduced using +N option in tail.
LOGFILE="filename.log"
tenminutesago=$(date --date='9000 minutes ago' +"%Y %b %e %H:%M") # matching pattern
lines=$(grep -n "$tenminutesago" $LOGFILE | grep -m 1 -oP "^[0-9]*" || echo "0") # lines after matching occurence
echo "$lines lines FOUND from the last X minutes"
let lines -eq 0 && tail -n +$lines $LOGFILE # last lines in file if lines is not 0
i have time logs in timestamp (epoch unix time) format :
1515365117236
1515365123162
1515365139963
i would like to convert it to a regular date like
2017-01-07 23:48:01
2017-01-07 23:48:02
2017-01-07 23:48:03
any ideas what approach would be the fastest?
cat ff1.csv | while read line ; do echo $line\;$(date -d +"%Y-%m-%d %H:%M:%S") ; done > somefile.csv
this takes awful lot of time and just appends the current time
Another approach that must be much faster , using printf of bash version >4.2 :
$ printf '%(datefmt)T\n' epoch
For datefmt you need a string accepted by strftime(3) - see man 3 strftime
Testing:
$ cat file10
1515365117236
1515365123162
1515365139963
$ printf '%(%F %H:%M:%S)T\n' $(cat file10)
49990-01-04 04:47:16
49990-01-04 06:26:02
49990-01-04 11:06:03
In this case , printf format string is:
%F Equivalent to %Y-%m-%d (the ISO 8601 date format). (C99)
%H The hour as a decimal number using a 24-hour clock (range 00 to 23).(Calculated from tm_hour.)
%M The minute as a decimal number (range 00 to 59). (Calculated from tm_min.)
%S The second as a decimal number (range 00 to 60). (The range is up to 60 to allow for occasional leap seconds.- Calculated from tm_sec.)
Update to remove milliseconds:
$ printf '%(%F %T)T\n' $(printf '%s/1000\n' $(<file10) |bc)
2018-01-08 00:45:17
2018-01-08 00:45:23
2018-01-08 00:45:39
The way to transform epoch to date is date -d #epochtime +format
An alternative way is to use date --file switch to read dates from a file directly.
$ cat file10
1515365117236
1515365123162
1515365139963
In order date to understand that these lines are epoch time you need to add # in the beginning of each line.
This can be done like bellow:
$ sed -i 's/^/#/g' file10 #caution - this will make changes in your file
$ date --file file10 +"%Y-%m-%d %H:%M:%S"
Alternativelly, you can do it on the fly without affecting the original file:
$ sed 's/^/#/g' file10 |date --file - +"%Y-%m-%d %H:%M:%S"
PS: in this case --file reads from - == stdin == pipe
In both cases, the result is
49990-01-04 04:47:16
49990-01-04 06:26:02
49990-01-04 11:06:03
PS: by the way, the timestamps you provide seems invalid, since it seems to refer at year 49990
Your input data aren't epoch unix time, it has miliseconds. If you wish to use any method on bash first you must convert to timestamp:
cat ff1.csv | while read LINE; do echo "#$(expr $LINE \/ 1000)" | date +"%Y-%m-%d %H:%M:%S" --file - ; done
First divide by 1000 to delete miliseconds parts, the rest is the same that explain George Vasiliou
I'm trying to convert strings, describing a time interval, to the corresponding number of seconds.
After some experimenting I figured out that I can use date like this:
soon=$(date -d '5 minutes 10 seconds' +%s); now=$(date +%s)
echo $(( $soon-$now ))
but I think there should be an easier way to convert strings like "5 minutes 10 seconds" to the corresponding number of seconds, in this example 310. Is there a way to do this in one command?
Note: although portability would be useful, it isn't my top priority.
You could start at epoch
date -d"1970-01-01 00:00:00 UTC 5 minutes 10 seconds" "+%s"
310
You could also easily sub in times
Time="1 day"
date -d"1970-01-01 00:00:00 UTC $Time" "+%s"
86400
There is one way to do it, without using date command in pure bash (for portability)
Assuming you just have an input string to convert "5 minutes 10 seconds" in a bash variable with a : de-limiter as below.
$ convertString="00:05:10"
$ IFS=: read -r hour minute second <<< "$convertString"
$ secondsValue=$(((hour * 60 + minute) * 60 + second))
$ printf "%s\n" "$secondsValue"
310
You can run the above commands directly on the command-line without the $ mark.
This will do (add the epoch 19700101):
$ date -ud '19700101 5 minutes 10 seconds' +%s
310
It is important to add a -u to avoid local time (and DST) effects.
$ TZ=America/Los_Angeles date -d '19700101 5 minutes 10 seconds' +%s
29110
Note that date could do some math:
$ date -ud '19700101 +5 minutes 10 seconds -47 seconds -1 min' +%s
203
The previous suggestions didn't work properly on alpine linux, so here's a small helper function that is POSIX compliant, is easy to use and also supports calculations (just as a side effect of the implementation).
The function always returns an integer based on the provided parameters.
$ durationToSeconds '<value>' '<fallback>'
$ durationToSeconds "1h 30m"
5400
$ durationToSeconds "$someemptyvar" 1h
3600
$ durationToSeconds "$someemptyvar" "1h 30m"
5400
# Calculations also work
$ durationToSeconds "1h * 3"
10800
$ durationToSeconds "1h - 1h"
0
# And also supports long forms for year, day, hour, minute, second
$ durationToSeconds "3 days 1 hour"
262800
# It's also case insensitive
$ durationToSeconds "3 Days"
259200
function durationToSeconds () {
set -f
normalize () { echo $1 | tr '[:upper:]' '[:lower:]' | tr -d "\"\\\'" | sed 's/years\{0,1\}/y/g; s/months\{0,1\}/m/g; s/days\{0,1\}/d/g; s/hours\{0,1\}/h/g; s/minutes\{0,1\}/m/g; s/min/m/g; s/seconds\{0,1\}/s/g; s/sec/s/g; s/ //g;'; }
local value=$(normalize "$1")
local fallback=$(normalize "$2")
echo $value | grep -v '^[-+*/0-9ydhms]\{0,30\}$' > /dev/null 2>&1
if [ $? -eq 0 ]
then
>&2 echo Invalid duration pattern \"$value\"
else
if [ "$value" = "" ]; then
[ "$fallback" != "" ] && durationToSeconds "$fallback"
else
sedtmpl () { echo "s/\([0-9]\+\)$1/(0\1 * $2)/g;"; }
local template="$(sedtmpl '\( \|$\)' 1) $(sedtmpl y '365 * 86400') $(sedtmpl d 86400) $(sedtmpl h 3600) $(sedtmpl m 60) $(sedtmpl s 1) s/) *(/) + (/g;"
echo $value | sed "$template" | bc
fi
fi
set +f
}
Edit : Yes. I developed for OP after comment and checked on Mac OS X, CentOS and Ubuntu. One liner, POSIX compliant command for converting "X minutes Y seconds" format to seconds. That was the question.
echo $(($(echo "5 minutes 10 seconds" | cut -c1-2)*60 + $(echo "5 minutes 10 seconds" | cut -c1-12 | awk '{print substr($0,11)}')))
OP told me via comment that he wants for "X minutes Y seconds" format not for HH:MM:SS format. The command with date and "+%s" is throwing error on (my) Mac. OP wanted to grab the numerical values from "X minutes Y seconds" format and convert it to seconds. First I extracted the minute in digit (take it as equation A) :
echo "5 minutes 10 seconds" | cut -c1-2)
then I extracted the seconds part (take it as equation B) :
echo "5 minutes 10 seconds" | cut -c1-12 | awk '{print substr($0,11)}'
Now multiply minute by 60 then add with the other :
echo $((equation A)*60) + (equation B))
OP should ask the others to check my developmental version (but working) of command before using it for automatic repeated usage like we do with cron on a production server.
If we want to run this on a log file with values in "X minutes Y seconds" format, we have to change echo "5 minutes 10 seconds" to cat file | ... like command. I kept a gist of it too if I or others ever need we can use it with cat to run on server log files with x minutes y seconds like log format.
Although off-topic (what I understood, question has not much to do with current time), this is not working for POSIX-compliant OS to get current time in seconds :
date -d "1970-01-01 00:00:00 UTC 5 minutes 10 seconds" "+%s"
It will throw error on MacOS X but work on most GNU/Linux distro. That +%s part will throw error on POSIX-compliant OS upon complicated usage. These commands are mostly suitable to get current time in seconds on POSIX compliant to any kind of unix like OS :
awk 'BEGIN{srand(); print srand()}'
perl -le 'print time'
If OP needs can extend it by generating current time in seconds and subtract. I hope it will help.
---- OLD Answer before EDIT ----
You can get the current time without that date -- echo | awk '{print systime();}' or wget -qO- http://www.timeapi.org/utc/now?\\s. Other way to convert time to second is echo "00:20:40.25" | awk -F: '{ print ($1 * 3600) + ($2 * 60) + $3 }'.
The example with printf shown in another answer is near perfect.
That thing you want is always needed by the basic utilities of GNU/Linux - gnu.org/../../../../../Setting-an-Alarm.html
Way to approach really depends how much foolproof way you need.
I am trying to compute time difference between dates formatted as below:
dd/mm/YY;hh:mm:ss;dd/mm/YY;hh:mm:ss (the first couple dd/mm/YY;hh:mm:ss points out the start date and the second couple is
the end date)
I want the output to be like this:
dd/mm/YY;hh:mm:ss;dd/mm/YY;hh:mm:ss;hh:mm:ss , where the added hh:mm:ss is the time difference between both dates.
Here is an example:
INPUT:
12/11/15;20:04:09;13/11/15;08:46:26
13/11/15;20:05:34;14/11/15;08:42:04
14/11/15;20:02:47;16/11/15;08:44:43
OUTPUT:
12/11/15;20:04:09;13/11/15;08:46:26;12:42:17
13/11/15;20:05:34;14/11/15;08:42:04;12:36:30
14/11/15;20:02:47;16/11/15;08:44:43;36:41:56
I've tried a lot of things with gsub, mktime and awk, in order to format dates, but nothing is efficient enough (too many operations to format and split).
Here is my attempt:
cat times.txt | awk -F';' '{gsub(/[/:]/," ",$0);d1=mktime("20"substr($1,7,2)" "substr($1,4,2)" "substr($1,1,2)" "$2);d2=mktime("20"substr($3,7,2)" "substr($3,4,2)" "substr($3,1,2)" "$4); print strftime("%H:%M:%S", d2-d1,1);}' > timestamps.txt
paste -d";" times.txt timestamps.txt
What do you suggest?
Thank you :)
You could try this and save some gsub and substr calls:
awk -F'[:;/]' '{d1=mktime("20"$3" "$2" "$1" "$4" "$5" "$6);
d2=mktime("20"$9" "$8" "$7" "$10" "$11" "$12);
delta = d2-d1
sec = delta%60
min = (delta - sec)%3600/60
hrs = int(delta/3600)
print $0";"(hrs < 10 ? "0"hrs : hrs)\
":"(min < 10 ? "0"min : min)\
":"(sec < 10 ? "0"sec : sec);}' time.txt
Since we cannot use strftime (tanks to Ed Morton), we have to handle the case that hours > 23 or hour/min/sec < 10 manually.
The above code outputs:
14/11/15;20:02:47;16/11/15;08:44:43;36:41:56
14/11/15;20:02:47;14/11/15;20:02:48;00:00:01
for the input
14/11/15;20:02:47;16/11/15;08:44:43
14/11/15;20:02:47;14/11/15;20:02:48
You cannot do this job robustly without mktime() as the time difference calculation needs to account for leap days, leap seconds, etc. I don't think you can do it any more efficiently than this:
$ cat tst.awk
BEGIN { FS="[/;:]" }
{
d1 = mktime("20"$3" "$2" "$1" "$4" "$5" "$6)
d2 = mktime("20"$9" "$8" "$7" "$10" "$11" "$12)
delta = d2 - d1
hrs = int(delta/3600)
min = int((delta - hrs*3600)/60)
sec = delta - (hrs*3600 + min*60)
printf "%s;%02d:%02d:%02d\n", $0, hrs, min, sec
}
$ awk -f tst.awk file
12/11/15;20:04:09;13/11/15;08:46:26;12:42:17
13/11/15;20:05:34;14/11/15;08:42:04;12:36:30
14/11/15;20:02:47;16/11/15;08:44:43;36:41:56
Note - you cannot use strftime() [alone] to calculate the hrs, mins, and secs because when your delta value is more than 1 day strftime() will return the hrs, mins, and secs associated with the time of day on the last day of that delta instead of the total number of hrs, mins, and secs associated with the entire delta.
What you're asking will be pretty tricky traditional awk.
Of course, gawk (GNU awk) supports mktime, but other awk implementations do not. But you can do this directly in bash, relying on the date command for your conversion. This solution uses BSD date (so it'll work in FreeBSD, NetBSD, OpenBSD, OSX, etc).
while IFS=\; read date1 time1 date2 time2; do
stamp1=$(date -j -f '%d/%m/%y %T' "$date1 $time1" '+%s')
stamp2=$(date -j -f '%d/%m/%y %T' "$date2 $time2" '+%s')
d=$((stamp2-stamp1))
printf '%s;%s;%s;%s;%02d:%02d:%02d\n' "$date1" "$time1" "$date2" "$time2" $(( (d/3600)%60)) $(( (d/60)%60 )) $((d%60))
done < dates.txt
Results:
12/11/15;20:04:09;13/11/15;08:46:26;12:42:17
13/11/15;20:05:34;14/11/15;08:42:04;12:36:30
14/11/15;20:02:47;16/11/15;08:44:43;36:41:56
Of course, if you're using a non-BSD OS, you may have to install bsddate (if it's available) to get this functionality, or figure out how to get something equivalent using the tools you have on hand.
I need to create a bash shell script starting with a day and then loop through each subsequent day formatting that output as %Y_%m_d
I figure I can submit a start day and then another param for the number of days.
My issue/question is how to set a DATE (that is not now) and then add a day.
so my input would be 2010_04_01 6
my output would be
2010_04_01
2010_04_02
2010_04_03
2010_04_04
2010_04_05
2010_04_06
[radical#home ~]$ cat a.sh
#!/bin/bash
START=`echo $1 | tr -d _`;
for (( c=0; c<$2; c++ ))
do
echo -n "`date --date="$START +$c day" +%Y_%m_%d` ";
done
Now if you call this script with your params it will return what you wanted:
[radical#home ~]$ ./a.sh 2010_04_01 6
2010_04_01 2010_04_02 2010_04_03 2010_04_04 2010_04_05 2010_04_06
Very basic bash script should be able to do this:
#!/bin/bash
start_date=20100501
num_days=5
for i in `seq 1 $num_days`
do
date=`date +%Y/%m/%d -d "${start_date}-${i} days"`
echo $date # Use this however you want!
done
Output:
2010/04/30
2010/04/29
2010/04/28
2010/04/27
2010/04/26
Note: NONE of the solutions here will work with OS X. You would need, for example, something like this:
date -v-1d +%Y%m%d
That would print out yesterday for you. Or with underscores of course:
date -v-1d +%Y_%m_%d
So taking that into account, you should be able to adjust some of the loops in these examples with this command instead. -v option will easily allow you to add or subtract days, minutes, seconds, years, months, etc. -v+24d would add 24 days. and so on.
#!/bin/bash
inputdate="${1//_/-}" # change underscores into dashes
for ((i=0; i<$2; i++))
do
date -d "$inputdate + $i day" "+%Y_%m_%d"
done
Very basic bash script should be able to do this.
Script:
#!/bin/bash
start_date=20100501
num_days=5
for i in seq 1 $num_days
do
date=date +%Y/%m/%d -d "${start_date}-${i} days"
echo $date # Use this however you want!
done
Output:
2010/04/30
2010/04/29
2010/04/28
2010/04/27
2010/04/26
You can also use cal, for example
YYYY=2014; MM=02; for d in $(cal $MM $YYYY | grep "^ *[0-9]"); do DD=$(printf "%02d" $d); echo $YYYY$MM$DD; done
(originally posted here on my commandlinefu account)
You can pass a date via command line option -d to GNU date handling multiple input formats:
http://www.gnu.org/software/coreutils/manual/coreutils.html#Date-input-formats
Pass starting date as command line argument or use current date:
underscore_date=${1:-$(date +%y_%m_%d)}
date=${underscore_date//_/-}
for days in $(seq 0 6);do
date -d "$date + $days days" +%Y_%m_%d;
done
you can use gawk
#!/bin/bash
DATE=$1
num=$2
awk -vd="$DATE" -vn="$num" 'BEGIN{
m=split(d,D,"_")
t=mktime(D[1]" "D[2]" "D[3]" 00 00 00")
print d
for(i=1;i<=n;i++){
t+=86400
print strftime("%Y_%m_%d",t)
}
}'
output
$ ./shell.sh 2010_04_01 6
2010_04_01
2010_04_02
2010_04_03
2010_04_04
2010_04_05
2010_04_06
2010_04_07