I am trying to write a bash script, to merge 24 files in a given day. The requirement changes during Day light saving time changes, where I get 23 or 25 files.
So, with further research I realized that day-light savings begins on the second Sunday of March(23) of every year and ends on first sunday of Novemeber(25).
I need more inputs to get second sunday in a given month to do the check of finding 23 or 25 files for March and November respectively.
Any inputs to help me with this will be really appreciated.
Thank you
Here is the sample code to find 24 files in a day-
if [ -z "$1" ];then
now=$(date -d "-1 days" +%Y-%m-%d);
else now=$1;
fi
load_date='load_date='$now
singlePath="$newPath/$load_date"
fileCount=$(hdfs dfs -ls -R $hdfsPath/$load_date/ | grep -E '^-' | wc -l)
path=$hdfsPath/$load_date
if [ $fileCount -eq 24 ]; then
echo "All files are available for "$load_date;
hadoop fs -cat $path/* | hadoop fs -put - $singlePath/messages.txt
else echo $fileCount" files are available for "$load_date"! Please note, few files are being missed";
fi
I wouldn't hardcode the dates of DST transistions. I would just count "how many hours did today have":
a "normal" day:
$ diff=$(( $(date -d now +%s) - $(date -d yesterday +%s) ))
$ echo $(( diff / 3600 ))
24
"spring forward"
$ diff=$(( $(date -d "2019-03-10 23:59:59" +%s) - $(date -d "2019-03-09 23:59:59" +%s) ))
$ echo $(( diff / 3600 ))
23
"fall back"
$ diff=$(( $(date -d "2019-11-03 23:59:59" +%s) - $(date -d "2019-11-02 23:59:59" +%s) ))
$ echo $(( diff / 3600 ))
25
One thing to note: since bash only does integer arithmetic, if the difference is not 86400 but 86399, you get:
$ echo $((86399 / 3600))
23
So, better to query yesterday's time first in the tiny-but-non-zero chance that the seconds tick over between the 2 date calls:
diff=$(( -$(date -d yesterday +%s) + $(date -d now +%s) ))
Here, $diff will be 86400 or 86401 (for non DST transition days), and dividing by 3600 will give 24 not 23.
Related
For example:
echo $(date) - $(date -r sample.txt)
Output:
90 days(for example)
Use %s seconds since 1970-01-01 00:00:00 UTC as in
echo $(expr $(date +%s) - $(date -r sample.txt +%s)) #!/bin/sh
echo $(($(date +%s) - $(date -r sample.txt +%s))) #/bin/bash
One more way
$ ls -l peter.txt
-rwxrw-r--+ 1 pppp qqqq 149 Dec 15 18:39 peter.txt
$ echo "(" $(date +%s) - $(date -r peter.txt +%s) ")/" 86400 | perl -nle ' print eval, " days" '
29.254537037037 days
$
I am looking for a bash script to who check time difference between two directory? But in $oldtime i can`t get only 20:09:24 i get this "20:09:24.660157390".
How can i get only hours:min only.
#!/bin/bash
oldtime=$(stat -c %y /mnt/dir1| awk '{print $2}' )
ctime=$(date | awk '{print $4}')
DIFF=$($oldtime-$ctime)
if [ $DIFF > 600 ]; then
echo "This directory have more that 10 min"
fi
You could shave off the extra suffix,
but that won't really help you.
20:09:24 - $ctime is not going to work,
because "20:09:24" is not a number.
You have several other syntax errors as well.
To calculate the difference between dates,
you need to work with seconds.
oldseconds=$(stat -c %Y /mnt/dir1)
newseconds=$(date +%s)
if (((newseconds - oldseconds) / 60)); then
echo "This directory have more than 10 min"
fi
Use epoch timestamp and date for converting to that format.
$ date +%H:%M -d #`stat -c %Y /tmp/`
19:19
But it's not recommended to use this format for comparing. Use unix timestamps instead:
if [ $(( $(date +%s) - $(stat -c %Y /tmp) )) -gt 600 ]; then
echo "This directory have more that 10 min"
fi
I am trying to write a bash script.
I am on Amazon-s3. I have a bucket1, which contains bucket2, bucket3, bucket4. The bucket 2 has b1, b2, b3, b4, b5 - in which each of them contain tar files being created with nd the file name.
When I do a s3cmd --recursive ls s3://bucket1/bucket2/ | awk '{print$1}' I print the date of all the tar files which are generated.
I want the date on the computer to compare it with that list and print the last 1 week of the files which have been created.
Format of date on s3 : year/month/date
Thank you.
You can use gnu date with the +%s option to get seconds, and then easily compare dates in whatever format, i.e.
> (( ( $(date +%s) - $(date -d "2014/03/08" +%s) ) < (7*24*60*60) )) \
&& echo "Less than week ago" \
|| echo "More than week ago"
More than week ago
> (( ( $(date +%s) - $(date -d "2014/04/03" +%s) ) < (7*24*60*60) )) \
&& echo "Less than week ago" \
|| echo "More than week ago"
Less than week ago
In short, I want something works like this:
When I input a date like 2012-12-27 and want to expand the date by
week(start with Monday), it
outputs:2012-12-24,2012-12-25,2012-12-26,2012-12-27,2012-12-28,2012-12-29,2012-12-30
When I input a date like 2012-12-27 and want to expand the date by month, it outputs:2012-12-01,2012-12-02 ... 2012-12-31
or, how can I group a bunch of dates by week? e.g. when I input2012-12-01,2012-12-02 ... 2012-12-31. It outputs:2012-12-01,2012-12-02|2012-12-03 ... 2012-12-09|2012-12-10 ... 2012-12-16|...|2012-12-31
I have no idea how to complete this, any clue may be helpful!
DAYSECS=86400 # seconds in a day
WEEKSECS=604800
echo "Expand on week:"
g_epoch=$(date +"%s" -d $1) # given date as seconds from epoch
g_dayno=$(date +"%u" -d $1) # given date as day of week
g_month=$(date +"%m" -d $1) # given month
g_year=$(date +"%Y" -d $1) # given year
s_epoch=$(($g_epoch - $DAYSECS * ($g_dayno - 1)))
e_epoch=$(($s_epoch + $WEEKSECS))
for etime in $(seq $s_epoch $DAYSECS $e_epoch); do
date +"%Y-%m-%d" -d "#$etime"
done
echo "Expand on month:"
s_epoch=$(date +"%s" -d "$g_year-$g_month-01")
e_epoch=$(($s_epoch + 4 * $WEEKSECS))
for etime in $(seq $s_epoch $DAYSECS $e_epoch); do
if [ $(date +"%m" -d "#$etime") -ne "$g_month" ]; then
break;
fi
date +"%Y-%m-%d" -d "#$etime"
done
The script below work out from #perreal, I leave it here because:
It shows the power of GNU date.
It makes #perreal 's idea more clear and more universal.
Thank you, perreal!
Here it is
dd="2012-12-27" # test date
DAYSECS=86400 # seconds in a day
echo "expand by week:"
s_epoch=$(date +%s -d "$dd -$(($(date +%u -d $dd) - 1)) day") # start date of the week
e_epoch=$(date +%s -d "1970-01-01 00:00:00 +0000 +${s_epoch} seconds +6 days") # end date of the week
for etime in $(seq $s_epoch $DAYSECS $e_epoch); do
date +"%Y-%m-%d" -d "#$etime";
done
echo "expand by month:"
s_epoch=$(date +%s -d "$dd -$(($(date +%d -d $dd) - 1)) day") # start date of the month
e_epoch=$(date +%s -d "1970-01-01 00:00:00 +0000 +${s_epoch} seconds +1 month -1 day") # end date of the month
for etime in $(seq $s_epoch $DAYSECS $e_epoch); do
date +"%Y-%m-%d" -d "#$etime";
done
If there are dates as 2010-06-01 and another as 2010-05-15
Using shell script or date command how to get the number of days between the two dates
Thanks..
Using only date and shell arithmetics:
echo $((($(date -d "2010-06-01" "+%s") - $(date -d "2010-05-15" "+%s")) / 86400))
There's a solution that almost works: use the %s date format of GNU date, which prints the number of seconds since 1970-01-01 00:00. These can be subtracted to find the time difference between two dates.
echo $(( ($(date -d 2010-06-01 +%s) - $(date -d 2010-05-15 +%s)) / 86400))
But the following displays 0 in some locations:
echo $((($(date -d 2010-03-29 +%s) - $(date -d 2010-03-28 +%s)) / 86400))
Because of daylight savings time, there are only 23 hours between those times. You need to add at least one hour (and at most 23) to be safe.
echo $((($(date -d 2010-03-29 +%s) - $(date -d 2010-03-28 +%s) + 43200) / 86400))
Or you can tell date to work in a timezone without DST.
echo $((($(date -u -d 2010-03-29 +%s) - $(date -u -d 2010-03-28 +%s)) / 86400))
(POSIX says to call the reference timezone is UTC, but it also says not to count leap seconds, so the number of seconds in a day is always exactly 86400 in a GMT+xx timezone.)
OSX date is different than GNU date. Got it working like this in OSX. This is not portable solution.
start_date=$(date -j -f "%Y-%m-%d" "2010-05-15" "+%s")
end_date=$(date -j -f "%Y-%m-%d" "2010-06-01" "+%s")
echo $(( ($end_date - $start_date) / (60 * 60 * 24) ))
Idea is still same as in the other answers. Convert dates to epoch time, subtract and convert result to days.
Got it
d1=`date +%s -d $1`
d2=`date +%s -d $2`
((diff_sec=d2-d1))
echo - | awk -v SECS=$diff_sec '{printf "Number of days : %d",SECS/(60*60*24)}'
thanks..
Gnu date knows %j to display the day in year:
echo $(($(date -d 2010-06-01 +%j) - $(date -d 2010-05-15 +%j)))
crossing year-boundaries will give wrong results, but since you gave fixed dates ...