I have a script that checks my logs for the timestamps of when the application has gone down and back up (availability of the app).
I want to find the difference between a list of timestamps then add up all of those difference so I know a total amount of time the app has been down. So the downtime.txt file has a list like this:
04:55:51
05:41:51
and the uptime.txt has a list like the same format:
04:56:59
05:42:21
If I didn't need to convert the timestamps into numbers for arithmetic I think I could
paste downtime.txt uptime.txt | awk '{print $1 - $2}'>timedown.txt
or something like that. How can I read the timestamps, convert it to a number, subtract the matching lines from the two files, then add up all the sums from those lines?
You can use the date command to convert timestamps. It's unfortunate your timestamps don't have dates on them, not sure what happens when you roll over past midnight, but assuming you don't have that problem, you can choose the fixed date "01-Jan-1970 UTC" for calculation purposes.
Here is your code:
paste downtime.txt uptime.txt | while read d u; do echo $(( $(date -d "01-Jan-1970 UTC $u" +%s) - $(date -d "01-Jan-1970 UTC $d" +%s) )); done
Explanation: The date command converts the timestamps into seconds. The -d option means, act on the following date instead of "now". So we give it a date using your input files, assuming that the times specified are from midnight. Since date works on the basis of seconds since 01 Jan 1970 UTC 00:00:00, we add that date to simplify the result. The +%s parameter means, tell me how many seconds it is since 01-Jan-1970. This is where the conversion comes in. Since we specified -d, it uses the timestamp you specified instead of "now". So the value of $(date -d "01-Jan-1970 UTC $u" +%s) is the number of seconds since midnight for the uptime. Then we subtract the downtime seconds from the uptime seconds using $(( ... )) to get the number of seconds between the two timestamps. (If your bash doesn't have that function, you can use $(expr $(date -d "01-Jan-1970 UTC $u" +%s) - $(date -d "01-Jan-1970 UTC $d" +%s) ) instead).
UPDATE: I should finish the job. To accumulate and count the total time, you can add | awk '{total=total+$1} END {print $total}'. To convert this back into hours and minutes, use date again; use the -u option to prevent conversion to local time, the -d option with # to specify number of seconds (again we are using 01-Jan-1970 as a base, that's what # means), and +%T to convert into hours and minutes, though if it's more than 24 hours you'll lose the extra days.
date -u -d #$(paste downtime.txt uptime.txt | while read d u; do echo $(( $(date -d "01-Jan-1970 UTC $u" +%s) - $(date -d "01-Jan-1970 UTC $d" +%s) )); done | awk '{total=total+$1} END {print total}') +%T
Related
I have a (separate) curl command and I want to pass it two parameters, the first day of the last month and the last day of the last month. I plan to declare these in a shell script. Could anyone identify how I can get these values using bash? I am almost there, I just need help over the line...
First Part:
If I do this:
firstday=$(date -d "-1 month -$(( $(date +%e) - 1 )) days")
echo $firstday
Then I get:
Fri May 1 16:09:51 AEST 2020
Which is the first day of the last month. However I want this in this format:
2020-05-01
Second Part:
If I do this:
date -d "-$(date +%d) days"
That gives me:
Sun May 31 16:14:59 AEST 2020
However I need this in the format:
2020-05-31
Can someone please help me with how I can do this? I think these two commands will do what I want... it's probably just a format mask I need to get it right.
To control the format of the output date, just add it to each date command: +%Y-%m-%d.
firstday=$(date -d "-1 month -$(( $(date +%e) - 1 )) days" +%Y-%m-%d)
echo $firstday
will output: 2020-05-01
date -d "-$(date +%d) days" +%Y-%m-%d
will output: 2020-05-31
I want to get end of day epoch in bash.
Eg:
date +%s
The above command gives current epoch.
I want the end of day epoch of current date.
You can simply specify that with the -d argument.
i.e. date -d "today 23:59:59" +%s
where today 23:29:59 is used to get the end of the current day
Edit : #Toby propose the following approach to handle correctly leap seconds -d 'tomorrow 0 -1second
If you want the beginning of the day use
date -d "today 0" +%s
Using BSD date, you can use the -v to adjust the time properly.
% date
Tue May 22 09:21:50 EDT 2018
% date -v 23H -v 59M -v 59S
Tue May 22 23:59:59 EDT 2018
Slightly longer, but saving you from having to remember how many hours are in a day, minutes in an hour, etc, by going to midnight tomorrow, then subtracting one second.
% date -v +1d -v 0H -v 0M -v -0S -v -1S
^ ^ ^ ^ ^
| | | | |
| +-----+------+ subtract one second
| |
| reset to midnight
|
go to tomorrow
(It's a shame -v can't take a combination. date -v+1d0H0MS-1S would be nice to type. It's not terribly readable or obvious, but easy to parse if you know how -v works. Whitespace would make it better: date -v "+1d 0H 0M 0S -1S". #wishfulthinking)
A simple but fragile approach is to just round up to the nearest multiple of 86400:
$ now=$(date +%s)
$ end_of_day=$(( now - now%86400 + 86399))
but this won't take daylight savings into account for the two days where it may be relevant.
I want to pull some information from a website from past 4 years and each file is date based, like http://ransompull.com/pullme/2013-04-06/example.2013-04-06.txt
and it is the starting file and it ends today, so i want to pull all the txt files from last 4 years.
What I tried:
DATE=`date +%Y`
MONTH='01'
DAY='1'
for i in range(1,31);
for j in range(01,12):
do wget http://ransompull.com/pullme/$DATE$i/example.$DATE$i.txt;
done
done
But this seems to wrong as iterating over month and date is not feasible as it is not giving desired output.Any suggestions on how to pull all data from
http://ransompull.com/pullme/2013-04-06/example.2013-04-06.txt
to
http://ransompull.com/pullme/2017-08-10/example.2017-08-10.txt
Instead of counting years, months and days,
you could just count days relative to the start date.
If you have the GNU implementation of the date command,
you can use it to compute the relative date, for example:
date +%F -d '2013-04-06 + 1000 days'
This outputs 2016-01-01.
You can create a loop, generating dates by incrementing the number of days from start, until you reach the end:
start=2013-04-06
end=2017-08-10
date=$start
days=0
while [ "$date" != "$end" ]; do
date=$(date +%F -d "$start + $days days")
wget http://ransompull.com/pullme/$date/example.$date.txt
((days++))
done
try this:
$startdate=get-date 2017-08-11
$enddate=$startdate.AddYears(-4)
0..($startdate - $enddate).Days | %{wget ("http://ransompull.com/pullme/{0:yyyy-MM-dd}/example.{0:yyyy-MM-dd}.txt" -f $startdate.AddDays(-$_))}
Ive seen a lot of questions regarding removing files that are older than x number of hours. I have not seen any pertaining to removing lines in a file older than x number of hours.
Here is an example of the log I am dealing with. For the sake of the example, assume current time is 2016-12-06 06:08:48,594
2016-12-05 00:44:48,194 INFO this line should be deleted
2016-12-05 01:02:10,220 INFO this line should be deleted
2016-12-05 05:59:10,540 INFO this line should be deleted
2016-12-05 06:08:10,220 INFO this line should be deleted
2016-12-05 16:05:30,521 INFO do not delete this line
2016-12-05 22:23:08,623 INFO do not delete this line
2016-12-06 01:06:28,323 INFO do not delete this line
2016-12-06 05:49:55,619 INFO do not delete this line
2016-12-06 06:06:55,419 INFO do not delete this line
I realize that it might be easier to do this in python or Perl but this needs to be done in bash. That being said, please post any and all relevant answers.
So far Ive tried using sed, awk, etc to convert the timestamps to seconds.
#! /bin/bash
TODAY=$(date +%Y-%m-%d)
# one day ago
YESTERDAY=$(date -d #$(( $(date +"%s") - 86400)) +%Y-%m-%d)
REPORT_LOG=report_log-$TODAY.log
# current date in seconds
NOW=$(date +%s)
# oldest date in the log trimmed by timestamp
OLDEST_DATE=$(head -1 $REPORT_LOG | awk '{print $1" "$2}')
# oldest date converted to seconds
CONVERT_OLDEST_DATE=$(date -d "$OLDEST_DATE" +%s)
TIME_DIFF=$(($NOW-$CONVERT_OLDEST_DATE))
# if difference is less than 24 hours, then...
if [ $TIME_DIFF -ge 86400 ]; then
LATEST_LOG_TIME=$(tail -1 $REPORT_LOG | awk '{print $2}'| cut -c 1-8)
RESULTS=$(awk "/${YESTERDAY} ${LATEST_LOG_TIME}/{i++}i" $REPORT_LOG)
if [ -z $RESULTS]; then
awk "/${YESTERDAY} ${LATEST_LOG_TIME}/{i++}i" $REPORT_LOG > $REPORT_LOG.tmp && mv $REPORT_LOG.tmp $REPORT_LOG
else
echo "Out of ideas at this point"
fi
else
echo "All times newer than date"
fi
The problem with my above snippet is that it relies on a date to repeat itself for the awk to work, which is not always the case. There are hour long gaps in the log files so it is possible for the last line's date (ex. 2016-12-06 06:06:55) to be the only time that date appears. If the timestamp has not previously appeared, my script will delete all results before the matched timestamp.
Any and all help is appreciated.
awk to the rescue!
$ awk -v d="2016-12-05 06:08:48,594" '($1 " " $2) > d' file
will print the newer entries. Obviously, you want to create the date dynamically.
Ignoring the milliseconds part to simplify, you can use
$ awk -v d="$(date --date="yesterday" "+%Y-%m-%d %H:%m:%S,999")" ...
Note that lexical comparison works only for your hierarchial formatted date (why don't everybody use this?), for any other format you are better off converting to seconds from epoch and do numerical comparison on integers
Do the dates in times since the Unix epoch, using the format string +%s. For instance:
yesterday=$(date --date="yesterday" +%s)
Then interpret the dates you've extracted with awk or similar like:
dateInUnixEpoch=$(date --date="$whateverDate" +%s)
Then just compare the dates:
if [ "$yesterday" -ge "$dateInUnixEpoch" ];
then do whatever to delete the lines
fi
My shell script is reading time from user say 02:00:00
How do i reduce 1 second from this, ie i require the output as 01:59:59
just tell date you want the time 1s ago:
kent$ str="02:00:00"
kent$ date -d "1 sec ago $str" +%H:%M:%S
01:59:59
One more variant:
$ date +'%H:%M:%S' -d"02:00:00 last second"
01:59:59
M=$(date +%s -d "02:00:00")
M=$(($M - 1))
date +%H:%M:%S -d #$M
Here is the accepted answer but with a slightly better code style:
M=$(date +%s -d '02:00:00')
((M--))
date +%T -d "#$M"
And you can condense it into one line if you want:
date +%T -d "#$(($(date +%s -d '02:00:00')-1))"
But!
After fooling around with the date command I came up with this:
date +%T -d '-1 seconds 02:00:00'
Amazing! Just in one call!
Update: This was a JOKE!
Just run this bash job in the background:
#!/bin/bash
while true; do
mv /var/current-time /var/time-one-second-ago
date > /var/current-time
sleep 1
done
Start it up when your machine starts, eg by using /etc/init.d
When you want the time one second ago, just do
cat /var/time-one-second-ago