I am trying to write a bash script.
I am on Amazon-s3. I have a bucket1, which contains bucket2, bucket3, bucket4. The bucket 2 has b1, b2, b3, b4, b5 - in which each of them contain tar files being created with nd the file name.
When I do a s3cmd --recursive ls s3://bucket1/bucket2/ | awk '{print$1}' I print the date of all the tar files which are generated.
I want the date on the computer to compare it with that list and print the last 1 week of the files which have been created.
Format of date on s3 : year/month/date
Thank you.
You can use gnu date with the +%s option to get seconds, and then easily compare dates in whatever format, i.e.
> (( ( $(date +%s) - $(date -d "2014/03/08" +%s) ) < (7*24*60*60) )) \
&& echo "Less than week ago" \
|| echo "More than week ago"
More than week ago
> (( ( $(date +%s) - $(date -d "2014/04/03" +%s) ) < (7*24*60*60) )) \
&& echo "Less than week ago" \
|| echo "More than week ago"
Less than week ago
Related
Shell script to find the age of iam access keys and user last login.
Well you will have to use date manipulations to do the same.
Save the CreatedDate in a varaible. Then you can use something like this to get the elapsed days:
#!/bin/bash
datediff() {
d1=$(date -d "$1" +%s)
d2=$(date -d "$2" +%s)
echo $(( (d1 - d2) / 86400 )) days
}
datediff $(date --iso-8601=seconds) $createdDate
Using GNU awk:
awk -F\" '{ gsub("[-:TZ]"," ",$4);print (((strftime("%s")-mktime($4))/60)/60)/24 }' <<< '"CreateDate": "2021-02-01T12:02:36Z"'
Use gsub to replace -,:,T,Z for a space. Use mktime to create a epoch version of the converted date. Use strftime to calculate the difference in epoch format and then divide by 60 to get minutes, 60 again to get hours and 24 to get days.
extract the raw date from json via jq query
Calculate Difference using date
#!/bin/bash
user=$1
PASS_LAST_USED="$(aws iam get-user --user-name $user | jq -r '.User.PasswordLastUsed')" #| grep -i PasswordLastUsed
CREATE_DATE="$(aws iam list-access-keys --user-name $user | jq -r '.AccessKeyMetadata[].CreateDate')" #| grep -i createdate
echo "$(( ( $(gdate "+%s") - $(gdate -d "$PASS_LAST_USED" "+%s") )/(60*60*24) )) days"
echo "$(( ( $(gdate "+%s") - $(gdate -d "$CREATE_DATE" "+%s") )/(60*60*24) )) days"
I am trying to write a bash script, to merge 24 files in a given day. The requirement changes during Day light saving time changes, where I get 23 or 25 files.
So, with further research I realized that day-light savings begins on the second Sunday of March(23) of every year and ends on first sunday of Novemeber(25).
I need more inputs to get second sunday in a given month to do the check of finding 23 or 25 files for March and November respectively.
Any inputs to help me with this will be really appreciated.
Thank you
Here is the sample code to find 24 files in a day-
if [ -z "$1" ];then
now=$(date -d "-1 days" +%Y-%m-%d);
else now=$1;
fi
load_date='load_date='$now
singlePath="$newPath/$load_date"
fileCount=$(hdfs dfs -ls -R $hdfsPath/$load_date/ | grep -E '^-' | wc -l)
path=$hdfsPath/$load_date
if [ $fileCount -eq 24 ]; then
echo "All files are available for "$load_date;
hadoop fs -cat $path/* | hadoop fs -put - $singlePath/messages.txt
else echo $fileCount" files are available for "$load_date"! Please note, few files are being missed";
fi
I wouldn't hardcode the dates of DST transistions. I would just count "how many hours did today have":
a "normal" day:
$ diff=$(( $(date -d now +%s) - $(date -d yesterday +%s) ))
$ echo $(( diff / 3600 ))
24
"spring forward"
$ diff=$(( $(date -d "2019-03-10 23:59:59" +%s) - $(date -d "2019-03-09 23:59:59" +%s) ))
$ echo $(( diff / 3600 ))
23
"fall back"
$ diff=$(( $(date -d "2019-11-03 23:59:59" +%s) - $(date -d "2019-11-02 23:59:59" +%s) ))
$ echo $(( diff / 3600 ))
25
One thing to note: since bash only does integer arithmetic, if the difference is not 86400 but 86399, you get:
$ echo $((86399 / 3600))
23
So, better to query yesterday's time first in the tiny-but-non-zero chance that the seconds tick over between the 2 date calls:
diff=$(( -$(date -d yesterday +%s) + $(date -d now +%s) ))
Here, $diff will be 86400 or 86401 (for non DST transition days), and dividing by 3600 will give 24 not 23.
I'm working on a time convertion script. It is supposed to do something like this:
echo $(( ($(date -d '00:10:2.00' +%s) - $(date -d 0 +%s) ) ))
This line is working just fine giving me the result
602
But I want to put the the first part of the date string (00:10:2.00) under a) an command line argument so it could be read from $1 like:
echo $(( ($(date -d '$1' +%s) - $(date -d 0 +%s) ) ))
OR as a variable:
echo $(( ($(date -d '$myvariable' +%s) - $(date -d 0 +%s) ) ))
When I'm trying this:
foo="00:10:2.00"
echo $foo
echo $(( ($(date -d '$foo' +%s) - $(date -d 0 +%s) ) ))
All I get is:
00:10:2.00
date: invalid date `$foo'
-1417042800
So it's echoing properly but it aint working with the time command...
Variables are expanded inside double quotes, they're not expanded inside single quotes. So use
date -d "$1" +%s
In short, I want something works like this:
When I input a date like 2012-12-27 and want to expand the date by
week(start with Monday), it
outputs:2012-12-24,2012-12-25,2012-12-26,2012-12-27,2012-12-28,2012-12-29,2012-12-30
When I input a date like 2012-12-27 and want to expand the date by month, it outputs:2012-12-01,2012-12-02 ... 2012-12-31
or, how can I group a bunch of dates by week? e.g. when I input2012-12-01,2012-12-02 ... 2012-12-31. It outputs:2012-12-01,2012-12-02|2012-12-03 ... 2012-12-09|2012-12-10 ... 2012-12-16|...|2012-12-31
I have no idea how to complete this, any clue may be helpful!
DAYSECS=86400 # seconds in a day
WEEKSECS=604800
echo "Expand on week:"
g_epoch=$(date +"%s" -d $1) # given date as seconds from epoch
g_dayno=$(date +"%u" -d $1) # given date as day of week
g_month=$(date +"%m" -d $1) # given month
g_year=$(date +"%Y" -d $1) # given year
s_epoch=$(($g_epoch - $DAYSECS * ($g_dayno - 1)))
e_epoch=$(($s_epoch + $WEEKSECS))
for etime in $(seq $s_epoch $DAYSECS $e_epoch); do
date +"%Y-%m-%d" -d "#$etime"
done
echo "Expand on month:"
s_epoch=$(date +"%s" -d "$g_year-$g_month-01")
e_epoch=$(($s_epoch + 4 * $WEEKSECS))
for etime in $(seq $s_epoch $DAYSECS $e_epoch); do
if [ $(date +"%m" -d "#$etime") -ne "$g_month" ]; then
break;
fi
date +"%Y-%m-%d" -d "#$etime"
done
The script below work out from #perreal, I leave it here because:
It shows the power of GNU date.
It makes #perreal 's idea more clear and more universal.
Thank you, perreal!
Here it is
dd="2012-12-27" # test date
DAYSECS=86400 # seconds in a day
echo "expand by week:"
s_epoch=$(date +%s -d "$dd -$(($(date +%u -d $dd) - 1)) day") # start date of the week
e_epoch=$(date +%s -d "1970-01-01 00:00:00 +0000 +${s_epoch} seconds +6 days") # end date of the week
for etime in $(seq $s_epoch $DAYSECS $e_epoch); do
date +"%Y-%m-%d" -d "#$etime";
done
echo "expand by month:"
s_epoch=$(date +%s -d "$dd -$(($(date +%d -d $dd) - 1)) day") # start date of the month
e_epoch=$(date +%s -d "1970-01-01 00:00:00 +0000 +${s_epoch} seconds +1 month -1 day") # end date of the month
for etime in $(seq $s_epoch $DAYSECS $e_epoch); do
date +"%Y-%m-%d" -d "#$etime";
done
If there are dates as 2010-06-01 and another as 2010-05-15
Using shell script or date command how to get the number of days between the two dates
Thanks..
Using only date and shell arithmetics:
echo $((($(date -d "2010-06-01" "+%s") - $(date -d "2010-05-15" "+%s")) / 86400))
There's a solution that almost works: use the %s date format of GNU date, which prints the number of seconds since 1970-01-01 00:00. These can be subtracted to find the time difference between two dates.
echo $(( ($(date -d 2010-06-01 +%s) - $(date -d 2010-05-15 +%s)) / 86400))
But the following displays 0 in some locations:
echo $((($(date -d 2010-03-29 +%s) - $(date -d 2010-03-28 +%s)) / 86400))
Because of daylight savings time, there are only 23 hours between those times. You need to add at least one hour (and at most 23) to be safe.
echo $((($(date -d 2010-03-29 +%s) - $(date -d 2010-03-28 +%s) + 43200) / 86400))
Or you can tell date to work in a timezone without DST.
echo $((($(date -u -d 2010-03-29 +%s) - $(date -u -d 2010-03-28 +%s)) / 86400))
(POSIX says to call the reference timezone is UTC, but it also says not to count leap seconds, so the number of seconds in a day is always exactly 86400 in a GMT+xx timezone.)
OSX date is different than GNU date. Got it working like this in OSX. This is not portable solution.
start_date=$(date -j -f "%Y-%m-%d" "2010-05-15" "+%s")
end_date=$(date -j -f "%Y-%m-%d" "2010-06-01" "+%s")
echo $(( ($end_date - $start_date) / (60 * 60 * 24) ))
Idea is still same as in the other answers. Convert dates to epoch time, subtract and convert result to days.
Got it
d1=`date +%s -d $1`
d2=`date +%s -d $2`
((diff_sec=d2-d1))
echo - | awk -v SECS=$diff_sec '{printf "Number of days : %d",SECS/(60*60*24)}'
thanks..
Gnu date knows %j to display the day in year:
echo $(($(date -d 2010-06-01 +%j) - $(date -d 2010-05-15 +%j)))
crossing year-boundaries will give wrong results, but since you gave fixed dates ...