Shell script to find the age of iam access keys and user last login.
Well you will have to use date manipulations to do the same.
Save the CreatedDate in a varaible. Then you can use something like this to get the elapsed days:
#!/bin/bash
datediff() {
d1=$(date -d "$1" +%s)
d2=$(date -d "$2" +%s)
echo $(( (d1 - d2) / 86400 )) days
}
datediff $(date --iso-8601=seconds) $createdDate
Using GNU awk:
awk -F\" '{ gsub("[-:TZ]"," ",$4);print (((strftime("%s")-mktime($4))/60)/60)/24 }' <<< '"CreateDate": "2021-02-01T12:02:36Z"'
Use gsub to replace -,:,T,Z for a space. Use mktime to create a epoch version of the converted date. Use strftime to calculate the difference in epoch format and then divide by 60 to get minutes, 60 again to get hours and 24 to get days.
extract the raw date from json via jq query
Calculate Difference using date
#!/bin/bash
user=$1
PASS_LAST_USED="$(aws iam get-user --user-name $user | jq -r '.User.PasswordLastUsed')" #| grep -i PasswordLastUsed
CREATE_DATE="$(aws iam list-access-keys --user-name $user | jq -r '.AccessKeyMetadata[].CreateDate')" #| grep -i createdate
echo "$(( ( $(gdate "+%s") - $(gdate -d "$PASS_LAST_USED" "+%s") )/(60*60*24) )) days"
echo "$(( ( $(gdate "+%s") - $(gdate -d "$CREATE_DATE" "+%s") )/(60*60*24) )) days"
Related
I have this script (which works fine) that will write all the date/time per second, from a start date/time till an end date/time to a file
while read line; do
FIRST_TIMESTAMP="20230109-05:00:01" #this is normally a variable that changes with each $line
LAST_TIMESTAMP="20230112-07:00:00" #this is normally a variable that changes with each $line
date=$FIRST_TIMESTAMP
while [[ $date < $LAST_TIMESTAMP || $date == $LAST_TIMESTAMP ]]; do
date2=$(echo $date |sed 's/ /-/g' |sed "s/^/'/g" |sed "s/$/', /g")
echo "$date2" >> "OUTPUTFOLDER/output_LABELS_$line"
date=$(date -d "$date +1 sec" +"%Y%m%d %H:%M:%S")
done
done < external_file
However this sometimes needs to run 10 times, and the start date/time and end date/time sometimes lies days apart.
Which makes the script take a long time to write all that data.
Now I am wondering if there is a faster way to do this.
Avoid using a separate date call for each date. In the next example I added a safety parameter maxloop, avoiding loosing resources when the dates are wrong.
#!/bin/bash
awkdates() {
maxloop=1000000
awk \
-v startdate="${first_timestamp:0:4} ${first_timestamp:4:2} ${first_timestamp:6:2} ${first_timestamp:9:2} ${first_timestamp:12:2} ${first_timestamp:15:2}" \
-v enddate="${last_timestamp:0:4} ${last_timestamp:4:2} ${last_timestamp:6:2} ${last_timestamp:9:2} ${last_timestamp:12:2} ${last_timestamp:15:2}" \
-v maxloop="${maxloop}" \
'BEGIN {
T1=mktime(startdate);
T2=mktime(enddate);
linenr=1;
while (T1 <= T2) {
printf("%s\n", strftime("%Y%m%d %H:%M:%S",T1));
T1+=1;
if (linenr++ > maxloop) break;
}
}'
}
mkdir -p OUTPUTFOLDER
while IFS= read -r line; do
first_timestamp="20230109-05:00:01" #this is normally a variable that changes with each $line
last_timestamp="20230112-07:00:00" #this is normally a variable that changes with each $line
awkdates >> "OUTPUTFOLDER/output_LABELS_$line"
done < <(printf "%s\n" "line1" "line2")
Using epoch time (+%s and #) with GNU date and GNU seq to
produce datetimes in ISO 8601 date format:
begin=$(date -ud '2023-01-12T00:00:00' +%s)
end=$(date -ud '2023-01-12T00:00:12' +%s)
seq -f "#%.0f" "$begin" 1 "$end" |
date -uf - -Isec
2023-01-12T00:00:00+00:00
2023-01-12T00:00:01+00:00
2023-01-12T00:00:02+00:00
2023-01-12T00:00:03+00:00
2023-01-12T00:00:04+00:00
2023-01-12T00:00:05+00:00
2023-01-12T00:00:06+00:00
2023-01-12T00:00:07+00:00
2023-01-12T00:00:08+00:00
2023-01-12T00:00:09+00:00
2023-01-12T00:00:10+00:00
2023-01-12T00:00:11+00:00
2023-01-12T00:00:12+00:00
if you're using macOS/BSD's date utility instead of the gnu one, the equivalent command to parse would be :
(bsd)date -uj -f '%FT%T' '2023-01-12T23:34:45' +%s
1673566485
...and the reverse process is using -r flag instead of -d, sans "#" prefix :
(bsd)date -uj -r '1673566485' -Iseconds
2023-01-12T23:34:45+00:00
(gnu)date -u -d '#1673566485' -Iseconds
2023-01-12T23:34:45+00:00
I am trying to write a bash script, to merge 24 files in a given day. The requirement changes during Day light saving time changes, where I get 23 or 25 files.
So, with further research I realized that day-light savings begins on the second Sunday of March(23) of every year and ends on first sunday of Novemeber(25).
I need more inputs to get second sunday in a given month to do the check of finding 23 or 25 files for March and November respectively.
Any inputs to help me with this will be really appreciated.
Thank you
Here is the sample code to find 24 files in a day-
if [ -z "$1" ];then
now=$(date -d "-1 days" +%Y-%m-%d);
else now=$1;
fi
load_date='load_date='$now
singlePath="$newPath/$load_date"
fileCount=$(hdfs dfs -ls -R $hdfsPath/$load_date/ | grep -E '^-' | wc -l)
path=$hdfsPath/$load_date
if [ $fileCount -eq 24 ]; then
echo "All files are available for "$load_date;
hadoop fs -cat $path/* | hadoop fs -put - $singlePath/messages.txt
else echo $fileCount" files are available for "$load_date"! Please note, few files are being missed";
fi
I wouldn't hardcode the dates of DST transistions. I would just count "how many hours did today have":
a "normal" day:
$ diff=$(( $(date -d now +%s) - $(date -d yesterday +%s) ))
$ echo $(( diff / 3600 ))
24
"spring forward"
$ diff=$(( $(date -d "2019-03-10 23:59:59" +%s) - $(date -d "2019-03-09 23:59:59" +%s) ))
$ echo $(( diff / 3600 ))
23
"fall back"
$ diff=$(( $(date -d "2019-11-03 23:59:59" +%s) - $(date -d "2019-11-02 23:59:59" +%s) ))
$ echo $(( diff / 3600 ))
25
One thing to note: since bash only does integer arithmetic, if the difference is not 86400 but 86399, you get:
$ echo $((86399 / 3600))
23
So, better to query yesterday's time first in the tiny-but-non-zero chance that the seconds tick over between the 2 date calls:
diff=$(( -$(date -d yesterday +%s) + $(date -d now +%s) ))
Here, $diff will be 86400 or 86401 (for non DST transition days), and dividing by 3600 will give 24 not 23.
I have two dates as follows:
2019-01-06 00:02:10 | END
2019-01-05 23:52:00 | START
How could I calculate and print the difference between START and END dates in seconds?
For above case I would like to get something like:
610
Assuming GNU implementation based OS, you can use date's option %s and -d to calculate the time difference in seconds using command substitution and arithmetic operations.
START="2019-01-05 23:52:00"
END="2019-01-06 00:02:10"
Time_diff_in_secs=$(($(date -d "$END" +%s) - $(date -d "$START" +%s)))
echo $Time_diff_in_secs
Output:
610
Hope this helps!!!
With bash and GNU date:
while read d t x x; do
[[ $x == "END" ]] && end="$d $t"
[[ $x == "START" ]] && start="$d $t"
done < file
end=$(date -u -d "$end" '+%s')
start=$(date -u -d "$start" '+%s')
diff=$(($end-$start))
echo "$diff"
Output:
610
See: man date
What you're asking for is difficult verging on impossible using pure bash. Bash doesn't have any date functions of its own. For date processing, most recommendations you'll get will be to use your operating system's date command, but the usage of this command varies by operating system.
In BSD (including macOS):
start="2019-01-05 23:52:00"; end="2019-01-06 00:02:10"
printf '%d\n' $(( $(date -j -f '%F %T' "$end" '+%s') - $(date -j -f '%F %T' "$start" '+%s') ))
In Linux, or anything using GNU date (possibly also Cygwin):
printf '%d\n' $(( $(date -d "$end" '+%s') - $(date -d "$start" '+%s') ))
And just for the fun of it, if you can't (or would prefer not to) use date for some reason, you might be able to get away with gawk:
gawk 'END{ print mktime(gensub(/[^0-9]/," ","g",end)) - mktime(gensub(/[^0-9]/," ","g",start)) }' start="$start" end="$end" /dev/null
The mktime() option parses a date string in almost exactly the format you're providing, making the math easy.
START="2019-01-05 23:52:00"
END="2019-01-06 00:02:10"
parse () {
local data=(`grep -oP '\d+' <<< "$1"`)
local y=$((${data[0]}*12*30*24*60*60))
local m=$((${data[1]}*30*24*60*60))
local d=$((${data[2]}*24*60*60))
local h=$((${data[3]}*60*60))
local mm=$((${data[4]}*60))
echo $((y+m+d+h+mm+${data[5]}))
}
START=$(parse "$START")
END=$(parse "$END")
echo $((END-START)) // OUTPUT: 610
Was trying to solve the same problem on a non-GNU OS, i.e. macOS. I couldn't apply any of the solutions above, although it inspired me to come up with the following solution. I am using some in-line Ruby from within my shell script, which should work out of the box on macOS.
START="2019-01-05 23:52:00"
END="2019-01-06 00:02:10"
SECONDS=$(ruby << RUBY
require 'date'
puts ((DateTime.parse('${END}') - DateTime.parse('${START}')) * 60 * 60 * 24).to_i
RUBY)
echo ${SECONDS}
# 610
Assuming the following time-formats:
MM:DD:YYYY hh:mm:ss:nn
How can I compute the difference between two times? I have tried the
following, but it seems to fail.
% Value1='08:27:2018 23:53:50:08'
% Value2='08:28:2018 00:00:08:89'
% echo "$(($(date -d "$Value2" '+%s') - $(date -d "$Value1" '+%s')))"
Update: as the OP changed his format
$ Value1='08:27:2018 23:53:50:08'
$ Value2='08:28:2018 00:00:08:89'
$ Value1=${Value1/://}; Value1=${Value1/://}; Value1=${Value1%:*}.${Value1##*:}
$ Value2=${Value2/://}; Value2=${Value2/://}; Value2=${Value2%:*}.${Value2##*:}
$ echo $(date -d "$Value2" '+%s.%N') - $(date -d "$Value1" '+%s.%N') | bc -l
378.810000000
So all you need to do is convert it to a format that date knows, this is MM/DD/YYYY
Original answer below:
The problem is that your date-time format is not really recognized.
The date format:
$ date -d "08272018"
date: invalid date ‘08272018’
The date command knows many formats, but it is hard for it to distinguish between YYYYMMDD, DDMMYYYY and MMDDYYYY. To be clear, what does 10021002 represent as a date?
In this simple format—without delimiters—date will recognize YYYYMMDD and YYMMDD
$ date -d "20180708"
Sun 8 Jul 00:00:00 UTC 2018
The time format:
The notation HH:MM:SS:ss is by far from standard. What does ss represent, ss 60th of a second? The normal notation would be more HH:MM:SS.sss This will be recognized.
$ date -d "23:53:50:08"
date: invalid date ‘23:53:50:08’
$ date -d "23:53:50.08" "+%a %d %b %T.%N %Z %Y"
Wed 12 Sep 23:53:50.080000000 UTC 2018
So if you get your date format correct, you already get a long way:
% Value1='20180827 23:53:50.08'
% Value2='20180828 00:00:08.89'
% echo "$(($(date -d "$Value2" '+%s') - $(date -d "$Value1" '+%s')))"
378
The sad thing is that we are missing our milliseconds for this you need floating point arithmetic and bash does not support it. But there are ways around that (How do I use floating-point division in bash?)
$ echo $(date -d "$Value2" '+%s.%N') - $(date -d "$Value1" '+%s.%N') | bc -l
378.810000000
I shortened variable names:
v1='08:27:2018 23:53:50:08'
v2='08:28:2018 00:00:08:89'
With GNU date, just stick to one safe input format, you can convert YYYY-MM-DD HH:MM:SS.NN to... anything another. (side note: I love freebsd date, where you can just specify -f option for strptime. I wish I could do that with GNU date). So we can:
v1_epoch=$(date -d "${v1:6:4}-${v1:0:2}-${v1:3:2} ${v1:11:2}:${v1:14:2}:${v1:17:2}.${v1:20}" +%s.%N)
v2_epoch=$(date -d "${v2:6:4}-${v2:0:2}-${v2:3:2} ${v2:11:2}:${v2:14:2}:${v2:17:2}.${v2:20}" +%s.%N)
It will get us values of seconds with nanosecond resolution since epoch time. Now we need to calc a difference, we need to use a tool like bc, cause bash does not support floating point calculations.
diff=$(printf "scale=9; %s - %s\n" "$v2_epoch" "$v1_epoch" | bc)
Now this represents the difference of time we need to represent in hours, minutes, seconds and miliseconds.
printf "%s.%.3s" $(date -d#"$diff" -u +'%H:%M:%S %N')
That's simple, but it will wrap around at 23 hours, so we can do better with bc. The rounding in bc is sometimes unexpected... you need to just get used to unexpected scale=0 lines:
printf "%02d:%02d:%02d.%03d\n" $(printf 'scale=11; a=%s; scale=0; h=a/3600; m=a%%3600/60; s=a%%60/1; ms=a*1000%%1000/1; h \n m \n s \n ms \n' '$diff' | bc -l)
A "neat" oneliner:
$ v1='08:27:2018 23:53:50:08'
$ v2='08:28:2018 00:00:08:89'
$ printf "%02d:%02d:%02d.%03d\n" $(printf 'scale=11; a=%s; scale=0; h=a/3600; m=a%%3600/60; s=a%%60/1; ms=a*1000%%1000/1; h \n m \n s \n ms \n' "$(printf "scale=9; %s - %s\n" "$(date -d "${v2:6:4}-${v2:0:2}-${v2:3:2} ${v2:11:2}:${v2:14:2}:${v2:17:2}.${v2:20}" +%s.%N)" "$(date -d "${v1:6:4}-${v1:0:2}-${v1:3:2} ${v1:11:2}:${v1:14:2}:${v1:17:2}.${v1:20}" +%s.%N)" | bc)" | bc -l)
I guess this could be even shortened with some here strings, but that just harms readability:
printf "%02d:%02d:%02d.%03d\n" $(<<<"scale=11; a=$(<<< "scale=9; $(date -d "${v2:6:4}-${v2:0:2}-${v2:3:2} ${v2:11:2}:${v2:14:2}:${v2:17:2}.${v2:20}" +%s.%N) - $(date -d "${v1:6:4}-${v1:0:2}-${v1:3:2} ${v1:11:2}:${v1:14:2}:${v1:17:2}.${v1:20}" +%s.%N)"$'\n' bc); scale=0; h=a/3600; m=a%3600/60; s=a%60/1; ms=a*1000%1000/1; h"$'\n'"m"$'\n'"s"$'\n'"ms"$'\n' bc -l)
Or you can create a function for conversion:
mydate_read() { date -d "${1:6:4}-${1:0:2}-${1:3:2} ${1:11:2}:${1:14:2}:${1:17:2}.${1:20}" +%s.%N; };
printf "%02d:%02d:%02d.%03d\n" $(<<<"scale=11; a=$(<<< "scale=9; $(mydate_read "$v2") - $(mydate_read "$v1")"$'\n' bc); scale=0; h=a/3600; m=a%3600/60; s=a%60/1; ms=a*1000%1000/1; h"$'\n'"m"$'\n'"s"$'\n'"ms"$'\n' bc -l)
I forgot, we can merge the two bc calls into one:
mydate_read() { date -d "${1:6:4}-${1:0:2}-${1:3:2} ${1:11:2}:${1:14:2}:${1:17:2}.${1:20}" +%s.%N; };
printf "%02d:%02d:%02d.%03d\n" $(printf 'scale=11; a=%s - %s; scale=0; h=a/3600; m=a%%3600/60; s=a%%60/1; ms=a*1000%%1000/1; h \n m \n s \n ms \n' "$(mydate_read "$v2")" "$(mydate_read "$v1")" | bc -l)
If there are dates as 2010-06-01 and another as 2010-05-15
Using shell script or date command how to get the number of days between the two dates
Thanks..
Using only date and shell arithmetics:
echo $((($(date -d "2010-06-01" "+%s") - $(date -d "2010-05-15" "+%s")) / 86400))
There's a solution that almost works: use the %s date format of GNU date, which prints the number of seconds since 1970-01-01 00:00. These can be subtracted to find the time difference between two dates.
echo $(( ($(date -d 2010-06-01 +%s) - $(date -d 2010-05-15 +%s)) / 86400))
But the following displays 0 in some locations:
echo $((($(date -d 2010-03-29 +%s) - $(date -d 2010-03-28 +%s)) / 86400))
Because of daylight savings time, there are only 23 hours between those times. You need to add at least one hour (and at most 23) to be safe.
echo $((($(date -d 2010-03-29 +%s) - $(date -d 2010-03-28 +%s) + 43200) / 86400))
Or you can tell date to work in a timezone without DST.
echo $((($(date -u -d 2010-03-29 +%s) - $(date -u -d 2010-03-28 +%s)) / 86400))
(POSIX says to call the reference timezone is UTC, but it also says not to count leap seconds, so the number of seconds in a day is always exactly 86400 in a GMT+xx timezone.)
OSX date is different than GNU date. Got it working like this in OSX. This is not portable solution.
start_date=$(date -j -f "%Y-%m-%d" "2010-05-15" "+%s")
end_date=$(date -j -f "%Y-%m-%d" "2010-06-01" "+%s")
echo $(( ($end_date - $start_date) / (60 * 60 * 24) ))
Idea is still same as in the other answers. Convert dates to epoch time, subtract and convert result to days.
Got it
d1=`date +%s -d $1`
d2=`date +%s -d $2`
((diff_sec=d2-d1))
echo - | awk -v SECS=$diff_sec '{printf "Number of days : %d",SECS/(60*60*24)}'
thanks..
Gnu date knows %j to display the day in year:
echo $(($(date -d 2010-06-01 +%j) - $(date -d 2010-05-15 +%j)))
crossing year-boundaries will give wrong results, but since you gave fixed dates ...